Automatic process for define row and field coverage

[joeldushouyu]

Hello,

I had some experience working on lawnmower project and wrote my own grid-base global planner(I think is renamed to planner server in newest documentation) in ros2. Personally, it is really exciting to see this field coverage in ros2.

I haven't actually test the demo on my laptop code. But from looking at the documentation and the launch demo, I have some questions and maybe some proposal for improvements.

From looking at the demo launch file regarding the pre-defined row pattern, it seems to me the pattern is stored from a .xml file.

While this makes sense to do for testing on a small area, I think it will be a pain to do it on a large area.
Thus, I am proposing to have the following additional function.

A package that can take the standard message type in ros2, like Odometry from /odometry/global and/or GPS from gps/fix and create the [.xml] file that defines the rows as mentioned above.
For example, we have a large area want to cover, but want to have special, irregular cutting patterns in some part of the areas. The user can define the large area, something like this demo, along with a ros2 bag that defines the irregular cutting patterns on specific areas. The package can then generate a unified .xml file for it.

Then possibly, user can re-edit and verified the result in rviz2, maybe related to this issue

I am not sure if this feature already existed in the repo or not. And I think this might potentially require new API change in F2C side?

[SteveMacenski]

That sounds easy enough. As I understand it, you're asking for the ability to create the XML based on the input topic which contains some definition of rows from an external source. I think that could be a simple stand-alone node like row_saver which takes in the rows message and outputs a file.

[joeldushouyu]

That sounds easy enough. As I understand it, you're asking for the ability to create the XML based on the input topic which contains some definition of rows from an external source.

That is mostly what I mean. the reason why I mention about a potential new API change is with the situation I have in mind.
Assuming the yellow shape defines the total area we want to cover and the blue lines are the rows created by the algorithm. On the other hand, the red box and the green rows is the customize behavior that user define through a bag or some like that.

Maybe I am wrong on this. But imaging the green rows are irregular and have complicated moving patterns, like moving in a circle, moving in a triangle.... The idea is that the green rows are in a completely different moving pattern from the blue rows. I wonder if I create a .XML file that specifies the yellow box, red box, and the green rows, will the F2C package work right out of the box to create the blue arrows and create the transition in/out path from blue rows to green rows and vice versa?

[SteveMacenski]

I don't think F2C will work for non-straight line segments like this, especially if they're not remotely colinear and have arbitrary curves.

[joeldushouyu]

Hmm, I see. If that is the case, maybe the user should use a different path planning algorithm, or maybe just a pure pursuit algorithm, on coverage area with arbitrary curves(the green part on the picture above).

But on the other hand, even if I define a .XML below, where blue arrow is filled by the F2C algorithm and the red and green arrows are defined by the customer, the package will not be able to automatically figure out a transition path from blue to red arrows, red to blue, and etc? In other words, the user should treat them as 3 separate* F2C field

[SteveMacenski]

the package will not be able to automatically figure out a transition path from blue to red arrows, red to blue, and etc? In other words, the user should treat them as 3 separate* F2C field

Likely, yes. I think its perhaps not the best solution to try to wedge everything into an F2C problem. You could have the main field F2C planning use and have the 'customer side missions' as a separate modality that doesn't have as many assumptions on it about rows and congruence. I'm not saying that you couldn't get it to work, but might be better to think about it in a couple of 'mode' scopes that each have their proper assumptions and optimal use-case than trying to find the 1-method-to-rule-them-all.

I'm also giving you armchair advice without fully understanding what you're doing and the constraints that you're under, so it could make sense to put this all into one F2C field problem - but it seems like your 'customer side missions' are alot more general than simply rows, so that leads me to think that that isn't the best direction to go

[joeldushouyu]

I think you are right on the point! I probably should try to think of those 'customer side missions' in different F2C missions or potentially use different algorithms.

But back to the point, are we still interested in adding the row_saver node into this repo as a way to save customize define rows?
And from the discussion above regarding on F2C does not work well with non-straight lines and curves, I imagine the node need to do some signal processing to filter out curves in the input signal. For example, if we record a bag that reflects the path in the demo, I guess the node should only preserve the blue straight lines and remove the green curves when turns.

[SteveMacenski]

are we still interested in adding the row_saver node into this repo as a way to save customize define rows?

If you think it would help, sure! We could do that in either python or C++ (I'd say python for ease of writing but C++ for not having to make a new package just for this utility). Only requirement I have is that it creates a rows file eq. to the example files you see in test/ that you've referenced above.

[joeldushouyu]

As I was setting up the docker environment for development, I rethought the application scenario and realized I probably should give more context to the issue and maybe you can give more valuable insight on the issue @SteveMacenski .

The problem:
I used to work on an autonomous lawnmower project for cutting grass on a golf court. One of the requirements for cutting grass on a golf court is the robot needs to follow the exact cutting pattern specified by the customer. Thus, we were hoping the robot to be able to cut in some similar pattern shown in the figure above.

Admittedly, we can achieve this goal by simply writing our own global planner that specified those cutting patterns with smaller grid sizes, like specifying each grid in the map to be 03*0.3 meters, which is part of the global planner that I mentioned earlier.

However, the downside of my approach is that I am defining the path on a 2D grid. This means it will take huge memory if I define the path to cover big fields.

But from looking at F2C documentation, it seems the solution addresses the issue of high memory usage since F2C is not a grid-base path planning algorithm. Thus, I was hoping that I could use the F2C package to define the areas that I want to cover by specificing the rows for cutting in a specific pattern on the golf court. And also, I was thinking on the worst-case scenario where the cutting pattern is noncolinear and have arbitrary curves, I can approximate curves in the path with small, intervals of straight lines. I guess the idea is like how we find arc-length in calculus.

Thus, as I rethought about it, I am not sure if it will be best for me to

1. write a new coverage planner that does that specifically
2. have this feature in F2C. The idea is to define different F2C fields for each corresponding cutting behavior.

Thanks you for being patient with me! Sorry this discussion is taking longer than it needs to be

[SteveMacenski]

It seems that you may want to have a few steps:

1. Use this work to generate your general coverage plan, in the absence of other customer-specific oddities. That lets you plan the broad space without needing a grid to accommodate your desire in not having the large memory consumption for what is effectively pre-ordained and immutable
2. Create a node that ingests that + these customer specific curves and setups for specialized patterns and stitches those into the route
3. Execute as you are today with your grid-based planner

The stitching part is the difficulty where I'd really need to dig into your application to give a specific suggestion. But in generalities, perhaps:

• Perhaps you know that the general switchbacks of F2C are the bulk of your space and time, customer specialized patterns are few and far between. In that case, you can use this to get the route to execute for the bulk of the field and when in the route you run into a pattern, you execute that, then continue back on the route
• Perhaps you know that you need to generate this once for a given location or client, so you can have someone hand stitch them as part of the robot setup period (with aided tooling, of course) like setting up the robot, dock, mapping, etc. Some level of annotation when setting up a new space is normal
• Perhaps you take the switchback patterns of this work as an unordered set of rows and you create your own routing algorithm that considers the rows & your customer curves to order the execution based on some criteria. For instance, if you had 200 rows and 10 customer curves, you could very realistically just brute force it to try every combination in a few seconds and store the result to use (or recompute each time on startup if the customer curves change alot). This isn't all that different from how F2Cv2 with heuristics works.
• etc.

[joeldushouyu]

First of all, thank you once again for the suggestions and the insight that you provided ! @SteveMacenski

• Perhaps you know that the general switchbacks of F2C are the bulk of your space and time, customer specialized patterns are few and far between.

As for my experience on the project, I think this might or might not be a valid assumption depending the requirement from the customers. Moreover, I think in general it might not be a good idea for this assumption if I want to go to the path of building the algorithm more general and available for community usage.

However, from your suggestions, I have a new thought on how the workflow might look like for supporting and integrating both customize define paths and paths planned by the F2C or other coverage algorithm.

1. Write the row_saver node that converts the bag file of the customized cutting pattern into rows and save into a file.
2. Mark the areas covered by the customized cutting pattern as obstacle/no entry area at the Route planning stage of F2C, or any general coverage planning algorithm, as shown in the documentation. Then, have the F2C to plan out the rows that cover the remaining area.
3. At the Path planning stage of F2C, this is where the stitching part come into the play.
   a. If the user only care about how the rows are being covered but now how the robot transitions from row to row, then we can use the idea below that you mentioned. Basically, throw the rows from both the customized define path and rows and run the algorithm provided in F2C at path planning to combine them together.

Perhaps you take the switchback patterns of this work as an unordered set of rows and you create your own routing algorithm that considers the rows & your customer curves to order the execution based on some criteria. For instance, if you had 200 rows and 10 customer curves, you could very realistically just brute force it to try every combination in a few seconds and store the result to use (or recompute each time on startup if the customer curves change alot). This isn't all that different from how F2Cv2 with heuristics works.

b. If the user do care about how the robot transitions from rows to rows, then they can either use the manual edition step in step 4(below) or write their own algorithm. For example, something like this issue?

4. On the idea of hand stitching everything together, as quoted below, it might be something that can be built in rviz, relating to this issue. Furthermore, we can also use this tool to manually view and verified the algorithm they building(mentioned in 3.b) is correct?

Perhaps you know that you need to generate this once for a given location or client, so you can have someone hand stitch them as part of the robot setup period

5. Then, save those into a new file and launch it.

Note: I believe this workflow of this will not cause problems to the issue regarding on "bypass static obstacles", since that should be something handled by the local planner(I believe is renamed to Controller server in the news doc for any dynamic obstacles. Static obstacles should be marked as forbidden area at route planning stage. And we can probably assume the customize behavior from the bag already avoids those static obstacles?

I am not sure if you and the community like the proposal above. Maybe I was being naive and made some bad assumptions on the step above.
But if you guys do find it to be useful for supporting customized define behavior into this repo, I can work on it and maybe submit a PR?

Note: I recently finished my undergraduate and moved on a different school for master'. Thus, I no longer work on the lawnmower project. But I believe I can obtain all the data I need through simulation and maybe make some mock GPS data base on odometry.

[SteveMacenski]

is this something you're still working on in your masters? I generally wouldn't recommend working on software if you don't have a way and need to test it. That's a recipe not to work out all the details fully since you aren't working with it on your day-to-day (then a mostly-functional-but-not-quite solution gets introduced and people try using it for their applications and have a suboptimal experience).

That workflow does seem generally reasonable. I'm not precisely sure about how you generalize step (1), but otherwise seems reasonable!

[joeldushouyu]

Unfortunately, robotics is not my focus for my master's degree. My master focus more on the computer architecture and hardware design of things. However, I am still providing remote support for the lawnmower project. it is just that I won't be there in person.. If everything goes well, I might be able to let them test and even getting a ros2 bag from for the customize cutting area.

For step 1, I have some initial thoughts on applying some signal processing techniques. Also, in term of distinguish Rows from the turning, I was thinking somehow calculate the curvature of the path and having a parameter that defines the threshold to filter out part of the path that has a sharp curve/turning. I once worked in a data analysis company that processes railroad data and worked on a project that shares some similar aspects in Python. I was hoping I could use similar techniques for step 1.

However, I do agree that it might be best for someone else in the community who can do full, extensive test to work on this, or even a collaboration. I guess we can leave this issue open and see if anyone else wants to pick it up in the future? @SteveMacenski

[SteveMacenski]

Why not use the actual rows that are used to create the bag file instead of extracting them from the bag files?

[joeldushouyu]

Why not use the actual rows that are used to create the bag file instead of extracting them from the bag files?

My idea is that the bag file is created by having someone manually drive the lawnmower to go through those customized coverage patterns. From my experience, the process of manually defining the rows by looking up on google map, finding the coordinate, etc is very time-consuming and might be inaccurate when covering a large area,

[SteveMacenski]

OK that works too. You could also just have someone press a button or something to annotate the start/end of a row so its less of a guessing game -- but I think what you describe should / could work!

[joeldushouyu]

OK that works too. You could also just have someone press a button or something to annotate the start/end of a row so its less of a guessing game -- but I think what you describe should / could work!

Hmm, that also might be a good, straightforward idea that might be easier to implement and play with. I guess at this point we should leave this issue open for someone else who finds it helpful and implement it, since I don't have the resources to do extensive testing?