Skip to content

Error when the input text can be confused with a relative path #6

Description

@mayeulk

problem

If the text string looks like a folder path, then fastText::language_identification() errors.
Example: "../.." (a text abbreviating "Continued on next page...")

minimal reproducible example

fastText::language_identification("../..", pre_trained_language_model_path =  "/path/to/lid.176.bin")

Erreur dans setnames(x, value) : 
  Impossible d'assigner 2 noms à une data.table de 1 colonne(s)
De plus : Message d'avis :
Dans data.table::fread(pth_res_out, header = F, stringsAsFactors = F,  :
  Le fichier '/tmp/RtmpMHsoaW/file16be26116117f.txt' a pour taille 0. Renvoi de data.table NULL.

(the error message seems identical to the English version at #3 (comment) )

brief description of the problem

I have text documents where "continued on next page" is abbreviated "../.." (not .../...). Those fail.
There are several ways to handle this.
The following work:
Prepend the text with a space (workaround):
fastText::language_identification(paste0(" ", "../..", pre_trained_language_model_path = "/path/to/lid.176.bin")

Change fastText::language_identification():

if (all(!(file.exists(input_obj)))) {

do not use file.exists but file_test(), as file.exists sees folders as files:
file.exists("../..")
[1] TRUE

file_test(op = "-f", x ="../..") # test whether this is a file (NOT a folder)
[1] FALSE

Third option: do not rely on automatic detection: add a parameter to the function,
specifying whether the input string should be considered as natural text, or as a file name.

Operating System (e.g., Windows10, Macintosh, Linux) and the R version

Linux Ubuntu 24.04
R 4.6.0

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingquestionFurther information is requestedtriage

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions