{"id":369,"date":"2016-01-27T11:06:24","date_gmt":"2016-01-27T10:06:24","guid":{"rendered":"http:\/\/www.francelabs.com\/blog\/?p=369"},"modified":"2016-01-27T11:07:17","modified_gmt":"2016-01-27T10:07:17","slug":"generating-big-data-sets-for-search-engines","status":"publish","type":"post","link":"https:\/\/www.francelabs.com\/blog\/generating-big-data-sets-for-search-engines\/","title":{"rendered":"Generating big data sets for search engines"},"content":{"rendered":"

NOTE: This is the English version. You will find the French version further down in this article.<\/p>\n

When proposing our expertise search, we are often asked to do performance evaluations on large datasets, for instance in Proof of Concepts. For a recent customer request, in order to gain time and to not use sensitive customer data, we have used log-synth, a random data generator<\/a> developed by Ted Dunning. We are describing here how to use log-synth in order to generate a 100.000 lines data set.<\/p>\n

The first step, which we don’t document here, is about downloading log-synth, unzipping it and building it with maven.<\/p>\n

<\/p>\n

The second step is about creating a schema that will be describing the way log-synth must generate each line. In our case, the goal is to generate log lines with the following format:<\/p>\n

{\"uuid\":\"41775b31-5435-4579-9803-99d78eb0512d\",\"server\":\"FL-01\",\"date\":\"2015-07-14\",\"nb_files\":53,\"status\":\"RUNNING\"}<\/code>
\nOff course, for each of these attributes, the values are picked randomly within a predefined set of values.
\nWe thus create the schema-francelabs.json file:<\/p>\n

[\r\n\t{"name": "uuid","class": "uuid"},\r\n\t{"name":"server", "class":"string", "dist":{"FL-01":1, "FL-02":1, "FL-03":1, "FL-04":1, "FL-05":1, "FL-06":1, "FL-07":1}},\r\n\t{"name": "date", "class": "date", "format": "yyyy-MM-dd", "start":"2015-01-01", "end":"2015-12-31"},\r\n\t{"name": "nb_files","class": "int","min": 1,"max": 100},\r\n\t{"name": "status", "class":"string", "dist":{"RUNNING":1, "OK":1, "ERROR":0.05}}\r\n]<\/pre>\n
For each attribute, we define its name using the tag “name” and its type thanks to the tag “class”. All the types managed by log-synth<\/a> as well as how to use them are listed and detailed in the log-synth documentation on github.<\/p>\n
Our schema is made of 5 attributes :<\/p>\n