Utility: FileLineSource

I have a small utility class which I use for reading in files with header lines, although it can be used for other line-record-oriented files.  Its primary advantage is that it encapsulates some error reporting and some basic setup.  It also does some minor exercising of the newer filesystem capabilities:

It has an interface, mainly to allow unit testing with an in-memory mock version:

class IFileLineSource

{

public:

  virtual ~IFileLineSource ();

  virtual void openSource () = 0;

  virtual bool getNextLine (std::string &outStr) = 0;

};

The actual concrete class is:

class FileLineSource : public IFileLineSource

{

public:

  FileLineSource (std::string_view inFilename,

  std::shared_ptr<Log::ILogger> inLogger,

  const bool inHasHeader = true);

  void openSource () override;

  bool getNextLine (std::string &outStr) override;

private:

  std::filesystem::path m_path;

  std::ifstream m_stream;

  bool m_finished;

  bool m_hasHeader;

  std::shared_ptr<Log::ILogger> m_logger;

};

with the functions defined as:

FileLineSource::FileLineSource(std::string_view inFilename,

                               std::shared_ptr<Log::ILogger> inLogger,

                               const bool inHasHeader):

    m_path(inFilename),

    m_finished(false), m_hasHeader(inHasHeader), m_logger(inLogger)

{

  if (!std::filesystem::exists(m_path))

    {

      m_logger->log(Log::Severity::Fatal, "File does not exist "

                                              + std::string(inFilename)

                                              + ": Exception thrown");

      throw std::runtime_error("File does not exist");

    }

}

void FileLineSource::openSource()

{

  m_stream.open(m_path);

  if (!m_stream.good())

    {

      m_logger->log(Log::Severity::Fatal,

                    "Unable to open file " + m_path.generic_string());

      throw std::runtime_error("Unable to open file");

    }

  if (m_hasHeader)

    {

      std::string s;

      std::getline(m_stream, s);

      if (s.empty())

m_finished = true;

    }

}

bool FileLineSource::getNextLine(std::string &outStr)

{

  if (m_finished)

    return false;

  std::getline(m_stream, outStr);

  if (outStr.empty())

    {

      m_finished = true;

      return false;

    }

  return true;

}

In the context of the LibraryThing project, this is used in reading in the lines from the downloaded file from the LibraryThing site.

There is a factory class as well, which can help separate logic out in come contexts:

class IFileLineSourceFactory

{

public:

  virtual ~IFileLineSourceFactory ();

  virtual std::unique_ptr<IFileLineSource> create () const = 0;

};

class FileLineSourceFactory : public IFileLineSourceFactory

{

public:

  FileLineSourceFactory (const std::string &inFilename,

                         std::shared_ptr<Log::ILogger> inLogger)

      : m_filename (inFilename), m_logger (inLogger)

  { }

  ~FileLineSourceFactory () override;

  std::unique_ptr<IFileLineSource> create () const override {   return std::make_unique<FileLineSource> (m_filename, m_logger); }

private:

  std::string m_filename;

  std::shared_ptr<Log::ILogger> m_logger;

};

Testing

Mock classes representing a file of specific kinds of records are going to be the very opposite of generic!  The mock class used in testing the memory record storage looks like:

class TestLineFactory : public JSBUtil::IFileLineSourceFactory

{

public:

  ~TestLineFactory() override {}

  std::unique_ptr<JSBUtil::IFileLineSource> create() const override

  {

    class TestLineSource : public JSBUtil::IFileLineSource

    {

    public:

      ~TestLineSource() override {}

      void openSource() override {}

      bool getNextLine(std::string &outStr) override

      {

        if (index == 0)

          {

            outStr

                = "13664096 The People's Anglican Missal (American "

                  "Edition) 5 London, England Society of SS. Peter "

                  "and Paul of Frank Gavin Liturgial "

                  "Foundation, Inc. (1946), Hardcover 1946 "

                  " The People's Anglican Missal American "

                  "Edition by Society of SS. Peter and Paul of London, "

                  "England (1946) Hardcover "

                  " "

                  " Liturgy, Anglicanism, "

                  "Anglo-Catholicism, Sacraments Your library "

                  "English English BX5947.M5 [] "

                  "Anglican Communion > Liturgy > Texts|Church of England > "

                  "Liturgy > Texts|Episcopal Church > Liturgy > "

                  "Texts|Episcopal Church. Holy "

                  "Communion|Liturgies|Missals 264.03 Anglican and American "

                  "P. E. ritual > Christian church and church work > Public "

                  "Worship; Ritual > Public worship; ritual > Religions "

                  " 1 amazon.com [2007-03-24] Anglican Book "

                  "Centre 477544 ";

            ++index;

            return true;

          }

        else if (index == 1)

          {

            outStr

                = "13665185 A Fire Upon The Deep 3 Vinge, "

                  "Vernor Tor Science Fiction "

                  "(1993), Reprint, Paperback 1992 "

                  " Box 11 A Fire Upon The Deep (Zones of Thought) by "

                  "Vernor Vinge (1993) Paperback 624 p.; 6.75 "

                  "inches 0.66 pounds 6.75 inches 1.36 "

                  "inches 4.2 inches 6.75 x 4.2 x 1.36 "

                  "inches 624 "

                  " Novels, SF/F, Singularity, Hugo Winner Your "

                  "library, Willowdale English English PS3572.I534 F57 "

                  "[0812515285] 0812515285, 9780812515282 Life on other "

                  "planets > Fiction|Science Fiction|Science fiction|science "

                  "fiction 813 American fiction > English (North "

                  "America) > Literature 1 "

                  "amazon.com [2007-03-24] 18501 "

                  " ";

            ++index;

            return true;

          }

        else if (index == 2)

          {

            outStr

                = "13665365 Caprice and Rondo 1 Dunnett, "

                  "Dorothy Michael Joseph "

                  "(1998), Paperback, 576 pages 1998 "

                  " Caprice and Rondo (The House of Niccolo, Book "

                  "7) by Dorothy Dunnett (1998) Paperback 576 p.; 9.13 "

                  "inches 1.68 pounds 9.13 inches 1.81 "

                  "inches 6.06 inches 9.13 x 6.06 x 1.81 "

                  "inches 576 "

                  " Historical Fiction, House of Niccolo, Novels, "

                  "Series, Scots Lit. Your library, Willowdale "

                  "English English PR6054.U56 C36 [0718140826] "

                  "0718140826, 9780718140823 Adventure and adventurers > "

                  "Poland > Fiction|Adventure stories|Bankers > Europe > "

                  "Fiction|Bankers > Fiction|Europe > History > 15th century "

                  "> Fiction|Fifteenth century > Fiction|Historical "

                  "Fiction|Historical fiction|Historical fiction. "

                  "gsafd|Merchants > Fiction|Middle East > History > 15th "

                  "century > Fiction|Poland > History > Casimir IV, 1447-1492 "

                  "> Fiction|Vander Poele, Nicholas (Fictitious character) > "

                  "Fiction|adventure stories|historical fiction 823.914 "

                  "1901-1999 > 1945-1999 > English fiction > English {except "

                  "North American} > Literature > Modern Period "

                  "1 Amazon.ca [2007-03-24] "

                  "1293143 ";

            ++index;

            return true;

          }

        else

          {

            outStr.clear();

            return false;

          }

      }

    private:

      int index = 0;

    };

    return std::make_unique<TestLineSource>();

  }

};

I find that in practice, although one can make data files part of one's unit test storage, it tends to work against the whole theme of repeatability in unit tests.  We separate out data into loadable files in "real" code precisely because we want to be able to modify the data without having to touch the source code.  With unit tests, we want invariability and repeatability.  This means taking one of two approaches with file-oriented code:

1) Do as above, providing a mock version which can provide all of the services except for the actual file access.

2) Write out a file during setup and then use that temporary file while the tests are run.  During the initial development of the unit tests you may want to make the file persistent to allow for simple debugging, but once your tests are complete it's a good idea to clean up the data files during teardown.

Comments

Post a Comment

Popular posts from this blog

Boundaries

Imagine that you have a well-designed application, possibly using dependency injection. It will almost always have a very thin layer visible in main() - capturing command-line parameters and environment variables, mainly[1], and then a call to an Application class which manages the rest of the initial logic - setting up resources such as logging, doing substantive validation of parameters, and creating and running the objects on the next level down which actually carry out the work of the application. This is simply following the SRP at a high level.
Read more

State Machines

Almost any program can be modelled as a state machine. I say "almost any" because it is, just, possible to think of a program which cannot easily be resolved into anything more than a one-state state machine. Consider a command interpreter with no support for loops or function definitions (like the old DOS command.com) and with no internal commands, only the ability to execute external commands. It loops forever and does only one thing for every line it reads.    If it does no error handling, the execution is a single line, a call to execve or one of its relatives. There are internal stages inside the C library call, but they are effectively invisible to the program.  However, it's not a very useful program. Neither is any version of HelloWorld. The echo application does have its uses. Once one gets beyond this, even simple programs have at least two states: setup and execute. A simple version of cp has to open two files for input and output before beginning to copy data....
Read more

Considerations on an Optimization

An application – actually, successive pair of applications – is meant to process (in the end) about three hundred or so entries to convert a simple markup into complex LaTeX markup. It also enhances the data with information drawn from a flat-file text database (i.e. literally a set of .txt files in a single directory). This application was originally written in rust, which is not my favourite language, but one worth exercising to a degree. The rust application covered a limited subdomain, and generally avoided unsafe rust code, although in some cases it had to use cell mechanisms which converted compile-time to runtime safety. The initial iteration had the text database as one large file, with some logic allowing for extended seeking through chapters. It soon became apparent, after extending my dataset to about twenty or so items, that this was going to result in an unacceptable runtime — it was taking longer to process than the later TeX processing took, and although in theory this w...
Read more