| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-10-06 | |||
| 13:49:07 | superdan | andreykurilin: such that they probably have very similar create times? | |
| 13:49:11 | andreykurilin | no | |
| 13:49:58 | superdan | andreykurilin: no meaning they were created one at a time in a client loop? | |
| 13:50:00 | andreykurilin | yes | |
| 13:50:08 | andreykurilin | sec | |
| 13:50:10 | superdan | and how many(ish)? | |
| 13:50:47 | andreykurilin | superdan: there are 2 instances which acre created in one time (~1 sec), but from different threads and with different names | |
| 13:51:08 | superdan | andreykurilin: so you're literally paging through two instances? | |
| 13:52:59 | andreykurilin | yes. just need to mention, that there are 2 cases and both failed. first one boot_and_list actions are performed twice in the same time. the second: list action performed once after both vms are booted | |
| 13:53:50 | superdan | andreykurilin: okay so limit=1 then? | |
| 13:54:13 | superdan | andreykurilin: and both instances are ACTIVE right? | |
| 13:57:06 | bauwser | zioproto: sorry, I have a huge internal backlog to do | |
| 13:59:49 | andreykurilin | superdan: so there are 2 cases. The shared logging relates to the first case, but the behaviour of nova the same and for the second case. Let me dedscribe it more details. there are 2 threads which perfroms boot_and_list actions. the listing is performed right after the vm become active. Both threads are using the same user and tenant | |
| 14:00:00 | zioproto | bauwser: no worries ! | |
| 14:00:35 | andreykurilin | superdan: in this case the first thread performs list action successfully (with using limit=-1 option of novaclient) and the second thread fails | |
| 14:00:59 | superdan | andreykurilin: and what does limit=-1 mean to novaclient? | |
| 14:01:15 | mriedem | page until there is nothing returned i think | |
| 14:01:20 | andreykurilin | yes | |
| 14:01:32 | superdan | right but with what limit to the api? | |
| 14:01:37 | superdan | no limit= default? | |
| 14:01:39 | andreykurilin | no limit | |
| 14:01:50 | mriedem | so default limit of 1000 | |
| 14:01:54 | andreykurilin | yes | |
| 14:02:13 | superdan | okay, so this really should get both instances in the first page, | |
| 14:02:17 | superdan | try another with result[-1] and get an empty page, yes? | |
| 14:03:05 | andreykurilin | `marker = result[-1]` gives the same page as previously with marker in it | |
| 14:03:25 | superdan | right, I was describing what _should_ be happening | |
| 14:03:48 | andreykurilin | yes | |
| 14:03:52 | superdan | okay | |
| 14:04:08 | superdan | I might have an idea of what is going on, but I need to do some experimentation | |
| 14:04:20 | superdan | andreykurilin: in the meantime, can you alter that loop a bit just to see if it helps? | |
| 14:04:22 | gibi | cburgess: hi! Is there any next step about https://blueprints.launchpad.net/nova/+spec/libvirt-virtio-set-queue-sizes I can look at / help with? | |
| 14:04:35 | andreykurilin | superdan: sure | |
| 14:04:44 | superdan | andreykurilin: can you set the sort_keys=['uuid'] | |
| 14:05:23 | superdan | although that really shouldn't matter since we're only iterating instances in a single cell db here | |
| 14:05:40 | superdan | andreykurilin: and I can throw up a nova patch you can depends-on right? | |
| 14:06:09 | superdan | andreykurilin: got a bug number for this yet? | |
| 14:06:42 | andreykurilin | superdan: yes, we can do depends-on to check the fix. no, I do not have a bug report | |
| 14:08:13 | andreykurilin | superdan: superdan: made a patch to check sort_keys, but based on the queue of the zuul the results will be in several hours | |
| 14:11:01 | andreykurilin | superdan: should I create a bug report? | |
| 14:11:14 | superdan | andreykurilin: yeah please create a bug and I'll some debugging | |
| 14:11:19 | superdan | sorry, I'm stuck on a call atm | |
| 14:11:24 | andreykurilin | thanks | |
| 14:21:57 | andreykurilin | superdan: https://bugs.launchpad.net/nova/+bug/1721791 | |
| 14:21:58 | openstack | Launchpad bug 1721791 in OpenStack Compute (nova) "Pagination of instances works incorrect" [Undecided,New] | |
| 14:22:34 | superdan | andreykurilin: thanks, I'm not sure how this is happening, but I'm really distracted on this call | |
| 14:22:39 | superdan | andreykurilin: are you going to be around for a while? | |
| 14:23:03 | andreykurilin | np, I'll planning to be there :) | |
| 14:30:20 | openstackgerrit | Merged openstack/nova master: Remove dest node allocations during live migration rollback https://review.openstack.org/507687 | |
| 14:33:38 | superdan | oooh, I might have a recreate | |
| 14:34:09 | cdent | must be because you’re super | |
| 14:35:15 | superdan | andreykurilin: do you do a regular unpaged list after the fail at all? | |
| 14:35:30 | superdan | andreykurilin: I kinda feel like one of the instances has to be in ERROR state, in cell0 to make this happen | |
| 14:38:57 | andreykurilin | superdan: while doing unpaged list, both instances are returned. After making a boot request, we are fetching the status of VM and do not continue until it become ACTIVE. Both VMs returned ACTIVE status | |
| 14:39:18 | andreykurilin | so I'm pretty sure that they are not in ERROR while listing | |
| 14:40:08 | superdan | andreykurilin: okay they should be sorted by created_at,id which is stable if you only have one database. Unless you have multiple cells here, or instances in cell0, I'm not sure how you could end up with unstable sort | |
| 14:41:59 | andreykurilin | superdan: it is dsvm job with a single node. it doesn't have any special configs | |
| 14:42:12 | superdan | yeah | |
| 14:42:26 | openstackgerrit | Dan Smith proposed openstack/nova master: WIP Always put 'uuid' into sort_keys for stabile instance lists https://review.openstack.org/510140 | |
| 14:42:35 | superdan | andreykurilin: can you try with this in place ^ ? | |
| 14:43:02 | superdan | if you revert the functional part of that change, the test added fails in the same way | |
| 14:43:53 | andreykurilin | ok, will make a depends on patch | |
| 14:51:34 | andreykurilin | superdan: btw, performance of list action is quite good. before I added a limit to the loop, debug messages flooded the log file by 10gb of text (until jenkins kicked the job by temout) :D | |
| 14:51:53 | superdan | andreykurilin: hah, cool | |
| 15:04:27 | superdan | andreykurilin: hmm, actually, that test isn't fully stable, so I need to keep working on it | |
| 15:05:08 | mriedem | fried_rice: merry friday https://review.openstack.org/#/c/488137/22 | |
| 15:05:15 | mriedem | i didn't -1, but i'm sort of inlined to | |
| 15:05:24 | fried_rice | mriedem ack, looking. | |
| 15:13:51 | cdent | ronlund: nobody wants to love on your doc fix? https://review.openstack.org/#/c/502168/ | |
| 15:17:28 | cdent | fried_rice: you still have https://review.openstack.org/#/c/499826/ in your mind? what do we need to do to resolve that? | |
| 15:17:43 | fried_rice | ... | |
| 15:18:11 | cdent | i hear that | |
| 15:18:21 | fried_rice | cdent I actually keep forgetting to put it on the "stuck reviews" list for the nova meetings. | |
| 15:18:45 | fried_rice | It really seems like overkill to put up a whole microversion for that change. | |
| 15:19:00 | cdent | yah | |
| 15:19:02 | fried_rice | But the process nazis would freak out if we slid it into some unrelated change that's doing a legit microversion bump. | |
| 15:19:47 | fried_rice | And I *totally* have no problem doing it without a microversion bump. If the rules forbid such a change, the rules are silly. | |
| 15:19:51 | openstackgerrit | Ildiko Vancsa proposed openstack/nova master: update live migration to use v3 cinder api https://review.openstack.org/463987 | |
| 15:20:00 | fried_rice | But I clearly don't get to make that call. | |
| 15:20:49 | fried_rice | cdent Makes it harder that we cut a release since the original splitup was done. | |
| 15:21:16 | fried_rice | you know, the one that didn't cut a new microversion when it changed the API in a similar (but more extensive) way than this. | |
| 15:22:04 | fried_rice | cdent Guess I'll add it to "stuck reviews" now while I'm thinking about it, and we can discuss it next Thursday. | |
| 15:22:18 | cdent | an astute plan | |
| 15:22:33 | fried_rice | cdent Thanks for the reminder | |
| 15:22:47 | cdent | was doing my weekly cruise of placement tagged bugs | |
| 15:25:52 | cdent | gibi: you seen https://bugs.launchpad.net/nova/+bug/1721652 ? references a change you made as the potential cause | |
| 15:25:53 | openstack | Launchpad bug 1721652 in OpenStack Compute (nova) "Evacuate cleanup fails at _delete_allocation_for_moved_instance" [Undecided,New] | |
| 15:27:33 | gibi | cdent: looking... | |
| 15:28:31 | openstackgerrit | Ed Leafe proposed openstack/nova master: Add alternate hosts https://review.openstack.org/486215 | |
| 15:28:32 | openstackgerrit | Ed Leafe proposed openstack/nova master: Return Selection objects from the scheduler driver https://review.openstack.org/495854 | |
| 15:28:32 | openstackgerrit | Ed Leafe proposed openstack/nova master: Add Selection objects https://review.openstack.org/499239 | |
| 15:28:33 | openstackgerrit | Ed Leafe proposed openstack/nova master: Change RPC for select_destinations() https://review.openstack.org/510159 | |
| 15:37:46 | gibi | cdent: I can confirm that bug based on looking at the code. It seems that the functional test did not catched it somehow | |
| 15:38:21 | cdent | gibi: cool, I figured you would know what was going on | |
| 15:39:24 | gibi | cdent: I felt save becuause of the functional coverage, but it seems we need a better test for it | |
| 15:39:32 | gibi | s/save/safe/ | |
| 15:42:11 | bauwser | leakypipes: maybe tracking all the changes for https://review.openstack.org/#/c/509025/ and above would be better if we have a specless BP ? | |
| 15:42:29 | bauwser | leakypipes: of course, not needing a spec | |
| 15:50:30 | ronlund | claudiub: want to send this in? https://review.openstack.org/#/c/509766/ | |
| 16:08:23 | finucannot | leakypipes superdan: Could you take a look at this? I don't think we need 'obj_make_compatible' functions because we're not transferring these objects over the wire, but it's gone to be sure to be sure https://review.openstack.org/#/c/508498/ | |