F.A.Q.

  1. What is Rawseeds? Who is it for? How can I contribute?

    Rawseeds is a project dedicated to the generation and publication of a high-quality Benchmarking Toolkit for robotics. Rawseeds has been designed to be useful both for researchers and companies. One of the key features of Rawseeds is the possibility for the users of the Toolkit to upload their own contributions to it and, doing so, to spread knowledge about their own work.

  2. Why does Rawseeds push towards the adoption of benchmarking procedures in robotics?

    Because we think that the development of standard benchmarking procedures in robotics is a key factor for the progress of the field. Such standards will allow an easy evaluation of the performance of new algorithms, a comparison between different approaches on objective grounds, and even a possibility to characterize commercial products.

  3. Why did you choose those sensors and not others?

    We tried to assemble a "representative sensor suite", defined as one including all sensor types usually adopted in mobile robotics. Each sensor type is represented by one or more items, chosen according to their diffusion in research and industrial applications.
    Of course we had to take into account the issues of overall mass and bulk, power consumption, possible interactions between sensors, and cost. Regarding the latter, we decided to avoid extremely costly devices: not only for budget issues (though of course we had to take them into account), but mainly because we wanted to concentrate on those sensor systems which are likely to be mounted on moderate-cost robots, including commercial ones.
    Descriptions of the sensors composing Rawseeds' sensor suite can be found here.

  4. Why do you provide the ground truth along with Rawseeds' datasets?

    Because ground truth is necessary to estimate the performance of Benchmark Solutions with the utmost accuracy. The availability of the ground truth for a data-gathering experiment allows comparison between the information reconstructed by applying a BS to sensor data and an objective description of the environment. The latter is obtained by means entirely separate from the robot's data-gathering sensors, thus avoiding any bias.

  5. Why do you think that “outdoor” and “mixed” Benchmark Problems are important?

    Most of the existing mobile robotics systems are indoor ones. Not because everyone feels that there are no outdoor applications, but simply because experimenting outdoors with robots is very difficult. Moreover, few outdoor or mixed datasets are available, so testing algorithms for outdoor applications is generally not possible without setting up a data gathering campaign... until Rawseeds.

  6. What is a “Deliverable”?

    A Deliverable is an official document of the project, i.e. one that the partners of the Rawseeds project are required to produce by contract. The Deliverables that you can find in this website are those which are useful for users.

  7. Is Rawseeds' Benchmarking Toolkit a tool for academics? How can it be useful for my company?

    The Toolkit is a powerful, high-quality instrument for anyone needing to evaluate and compare algorithms for localization, mapping, or SLAM. This, of course, includes both researchers and companies working in the robotics field... or testing the waters before entering it. The fact that the Toolkit is free, while experimenting with real robots is extremely costly, could also be considered as an advantage :-)

  8. Why should I (or my company) publish my algorithms on the Rawseeds website?

    Because in that way you give to them high visibility. If you are a researcher, this means that other colleagues could chose to adopt your approach towards the problems you are studying, and thay your contribution is acknowledged more widely. If you are the R&D head of a company, having the description and the results of your algorithms on Rawseeds means that potential buyers (or business partners) know that your products are good enough to be compared - on objective grounds - with the state of the art.

  9. Why do you require that my results are set in the form of a Benchmark Solution to publish them?

    Because, by expressing your results in that form, you make them comparable with those from other groups. Each Benchmark Problem includes a description of the procedures to evaluate the output of the Benchmark Solutions to that problem: this means that the results (obtained by applying the above procedures) of all the BSs associated to a specific BP are immediately comparable.

  10. How can you guarantee that people don't include fake results in their Benchmark Solutions?

    Because good results catch the attention of other people, and in particular of researchers, who will try to get similar performance by adopting similar approaches. If a good result is found to be not replicable, the author of the BS including such result will be soon challenged to explain how that can be. If such explanations are not good ones (and they can't be if the result is a fake), the forgery will be exposed and its author will lose any credibility through the whole robotics community. We think that this is a strong enough deterrent to avoid any cheating.

  11. You say that I can contribute my own Benchmark Solutions. Can I send you new datasets too?

    You can. But before your own datasets can be considered for publication, they must be subjected to data validation procedures similar to those that Rawseeds' data have been subjected to.

  12. I have an idea for a new Benchmark Problem. Can I submit it?

    Yes, you can. But please notice that - even if it is a good idea! - it will be taken into consideration only if it fits well with Rawseeds' strategy for the promotion of benchmarking in robotics.

  13. How is the project going?

    Please read here to know.

  14. I want to download your data. But in campus, torrents are not allowed to use. How can I download your data in another way?

    Unfortunately, distribution via BitTorrent is the only available method. So you should ask your IT administrators for a temporary exception, or ask the academic staff to do the download for you (they may not be subject to the same constraints). Otherwise, you will have to download our data from outside the campus.

  15. If I use the Rawseeds data in my paper, how should I cite it?

    You should add to your paper the following references:

    Andrea Bonarini, Wolfram Burgard, Giulio Fontana, Matteo Matteucci, Domenico Giorgio Sorrenti and Juan Domingo Tardos. RAWSEEDS: Robotics Advancement through Web-publishing of Sensorial and Elaborated Extensive Data Sets. In proceedings of IROS'06 Workshop on Benchmarks in Robotics Research, 2006.

    @inproceedings{Bonarini_2006_IROS,author={Andrea Bonarini, Wolfram Burgard, Giulio Fontana, Matteo Matteucci, Domenico Giorgio Sorrenti and Juan Domingo Tardos},
    title={RAWSEEDS: Robotics Advancement through Web-publishing of Sensorial and Elaborated Extensive Data Sets},
    booktitle={In proceedings of IROS'06 Workshop on Benchmarks in Robotics Research},
    year={2006},
    url={http://www.robot.uji.es/EURON/en/iros06.htm},
    }

    Simone Ceriani, Giulio Fontana, Alessandro Giusti, Daniele Marzorati, Matteo Matteucci, Davide Migliore, Davide Rizzi, Domenico G Sorrenti, Pierluigi Taddei. Rawseeds ground truth collection systems for indoor self-localization and mapping. Autonomous Robots 27(4):353--371, 2009.

    @article{Ceriani_2009_AutonomousRobots,author={Simone Ceriani, Giulio Fontana, Alessandro Giusti, Daniele Marzorati, Matteo Matteucci, Davide Migliore, Davide Rizzi, Domenico G. Sorrenti, Pierluigi Taddei},
    title={Rawseeds ground truth collection systems for indoor self-localization and mapping},
    journal={Autonomous Robots},
    year={2009},
    volume={27},
    number={4},
    pages={353--371},
    doi={10.1007/s10514-009-9156-5},
    url={http://www.springerlink.com/content/k924032g72818h53/},
    }

  16. What is the format of ground truth files?

    Each dataset includes a file called _FileFormats*.tar.torrent (where * is a string which can differ from one dataset to the other). Such file contains a description of the formats used for all data files.


If the question you were thinking of is not in the FAQ list, what about… submitting it to us?

Question:
Email: 1

1 - Notification of when your question has been answered. (Optional)