| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-10-06 | |||
| 12:44:27 | fried_rice | I'm inclined to think of an aggregate as kind of a special case of a trait. | |
| 12:45:48 | cdent | I’m not entirely following that logic? | |
| 12:45:50 | fried_rice | In which case, aggregates should propagate downward (from parent to child) like traits. | |
| 12:46:21 | fried_rice | Yeah, it's not fully formed in my bean. Something like... | |
| 12:46:25 | leakypipes | fried_rice: ack | |
| 12:47:57 | fried_rice | An aggregate is an "implicit trait" that you don't actually ask for, but that we kinda add to the request as we go along. That is, once we pick a RP for one piece of the request, we implicitly add its invisible-aggregate-trait for purposes of the rest of the request, so we only get the rest of the inventory from RPs with that same invisible-aggregate-trait. | |
| 12:48:06 | fried_rice | leakypipes What were you acking? | |
| 12:48:10 | leakypipes | fried_rice: ack on thing above about when you use a numbered deal, that means same provider. | |
| 12:48:19 | fried_rice | leakypipes Cool | |
| 12:48:33 | leakypipes | fried_rice: aggregates don't have traits. aggregates are nothing but grouping mechanisms | |
| 12:49:14 | fried_rice | yah, I get that, but am I off base logically thinking of them as described ^x4? | |
| 12:51:49 | cdent | leakypipes: am I right that the nested stack awaits resolutino of the no-orm stack (want to mention that in the rp update if it is in fact true)? | |
| 12:54:27 | fried_rice | cdent That's what I have been led to understand. | |
| 12:54:37 | cdent | ✔ | |
| 12:54:48 | fried_rice | It also occurred to me that I hadn't seen the code on the placement side that handles tree-ness for GET /allocation_candidates | |
| 12:55:04 | fried_rice | Like, the SQL magic that does downward trait propagation and suchlike. | |
| 12:55:17 | cdent | doesn’t exist yet as far as I’m aware? | |
| 12:55:17 | fried_rice | At least, I don't *think* I've seen that. | |
| 12:55:24 | fried_rice | okay. | |
| 12:55:54 | cdent | parts of it will be in https://review.openstack.org/#/q/topic:bp/nested-resource-providers+status:open | |
| 12:56:28 | fried_rice | cdent Yeah, I reviewed that stack and don't recall seeing the bits I'm talking about. | |
| 13:02:34 | openstackgerrit | OpenStack Proposal Bot proposed openstack/os-vif stable/ocata: Updated from global requirements https://review.openstack.org/490256 | |
| 13:14:54 | openstackgerrit | Matthew Booth proposed openstack/nova-specs master: Virtual instance rescue with stable disk devices https://review.openstack.org/510106 | |
| 13:15:30 | mdbooth | lyarwood: ^^^ | |
| 13:16:04 | mdbooth | lyarwood: Although it's your spec with only request REST API changes | |
| 13:18:46 | cdent | https://thebritishdrea.com/?text=Nested+providers+will+fix+That | |
| 13:24:53 | mdbooth | cdent: Lol | |
| 13:40:28 | andreykurilin | mriedem: hi! Do you know anything merged recently which could affect pagination? | |
| 13:40:45 | superdan | andreykurilin: pagination of instances? | |
| 13:40:49 | andreykurilin | yes | |
| 13:40:55 | superdan | andreykurilin: what are you seeing? | |
| 13:41:01 | mriedem | duh duh duh | |
| 13:41:12 | andreykurilin | superdan: marker doesn't work in some cases | |
| 13:41:49 | superdan | andreykurilin: heh, any more detail than that? :) | |
| 13:42:01 | mriedem | andreykurilin: have a failed job log? | |
| 13:42:03 | andreykurilin | superdan: sure, just need to collect links:) | |
| 13:42:03 | mriedem | novaclient or rally? | |
| 13:42:14 | andreykurilin | give me a sec | |
| 13:44:38 | andreykurilin | rally gates are failing due to an issue with pagination. we use limit=-1 option from novaclient. It is designed to make an inf loop changing the marker until the response will include an empty list. For some reasons API ignores the market in some cases. I copy-pasted the code from novaclient and added some debug messages | |
| 13:44:53 | andreykurilin | here is a ok execution - http://logs.openstack.org/83/509783/3/check/gate-rally-dsvm-neutron-existing-users-rally/1a480ab/console.html#_2017-10-05_23_02_16_892620 | |
| 13:45:12 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Add error notification for instance.interface_attach https://review.openstack.org/506643 | |
| 13:45:27 | andreykurilin | and just after several seconds, there is one more execution (another iteration of the workload) and it stucks | |
| 13:45:37 | andreykurilin | http://logs.openstack.org/83/509783/3/check/gate-rally-dsvm-neutron-existing-users-rally/1a480ab/console.html#_2017-10-05_23_02_19_441231 | |
| 13:45:52 | andreykurilin | here is a code which I'm using for dedbugging purpose - https://review.openstack.org/#/c/509783/3/rally/plugins/openstack/scenarios/nova/utils.py | |
| 13:46:13 | superdan | andreykurilin: is that implying that you get a page with a marker and you get back a page with the marker in it? | |
| 13:46:16 | andreykurilin | it is equal to what we have in novaclient but with some debug message as I already mentione | |
| 13:46:38 | andreykurilin | superdan: yes | |
| 13:46:42 | andreykurilin | i think so | |
| 13:47:01 | superdan | hrm | |
| 13:47:06 | andreykurilin | but it happens not regulary. In some cases the page includes the marker, in others - no | |
| 13:47:36 | superdan | what sort key are you using? | |
| 13:47:51 | andreykurilin | superdan: due to migration to Zull v3 I cannot say when it had happend actually, but can assume that 2 days ago | |
| 13:47:58 | andreykurilin | no sort keys | |
| 13:48:14 | superdan | andreykurilin: yeah I know what changed, so no question there | |
| 13:48:16 | andreykurilin | suyuperdan: here is a query http://logs.openstack.org/83/509783/3/check/gate-rally-dsvm-neutron-existing-users-rally/1a480ab/console.html#_2017-10-05_23_02_19_452283 | |
| 13:48:18 | superdan | andreykurilin: okay so default sort | |
| 13:48:29 | andreykurilin | just marker in query, nothing more | |
| 13:48:56 | superdan | andreykurilin: are these instances from a num_instances=N type create operation? | |
| 13:49:07 | superdan | andreykurilin: such that they probably have very similar create times? | |
| 13:49:11 | andreykurilin | no | |
| 13:49:58 | superdan | andreykurilin: no meaning they were created one at a time in a client loop? | |
| 13:50:00 | andreykurilin | yes | |
| 13:50:08 | andreykurilin | sec | |
| 13:50:10 | superdan | and how many(ish)? | |
| 13:50:47 | andreykurilin | superdan: there are 2 instances which acre created in one time (~1 sec), but from different threads and with different names | |
| 13:51:08 | superdan | andreykurilin: so you're literally paging through two instances? | |
| 13:52:59 | andreykurilin | yes. just need to mention, that there are 2 cases and both failed. first one boot_and_list actions are performed twice in the same time. the second: list action performed once after both vms are booted | |
| 13:53:50 | superdan | andreykurilin: okay so limit=1 then? | |
| 13:54:13 | superdan | andreykurilin: and both instances are ACTIVE right? | |
| 13:57:06 | bauwser | zioproto: sorry, I have a huge internal backlog to do | |
| 13:59:49 | andreykurilin | superdan: so there are 2 cases. The shared logging relates to the first case, but the behaviour of nova the same and for the second case. Let me dedscribe it more details. there are 2 threads which perfroms boot_and_list actions. the listing is performed right after the vm become active. Both threads are using the same user and tenant | |
| 14:00:00 | zioproto | bauwser: no worries ! | |
| 14:00:35 | andreykurilin | superdan: in this case the first thread performs list action successfully (with using limit=-1 option of novaclient) and the second thread fails | |
| 14:00:59 | superdan | andreykurilin: and what does limit=-1 mean to novaclient? | |
| 14:01:15 | mriedem | page until there is nothing returned i think | |
| 14:01:20 | andreykurilin | yes | |
| 14:01:32 | superdan | right but with what limit to the api? | |
| 14:01:37 | superdan | no limit= default? | |
| 14:01:39 | andreykurilin | no limit | |
| 14:01:50 | mriedem | so default limit of 1000 | |
| 14:01:54 | andreykurilin | yes | |
| 14:02:13 | superdan | okay, so this really should get both instances in the first page, | |
| 14:02:17 | superdan | try another with result[-1] and get an empty page, yes? | |
| 14:03:05 | andreykurilin | `marker = result[-1]` gives the same page as previously with marker in it | |
| 14:03:25 | superdan | right, I was describing what _should_ be happening | |
| 14:03:48 | andreykurilin | yes | |
| 14:03:52 | superdan | okay | |
| 14:04:08 | superdan | I might have an idea of what is going on, but I need to do some experimentation | |
| 14:04:20 | superdan | andreykurilin: in the meantime, can you alter that loop a bit just to see if it helps? | |
| 14:04:22 | gibi | cburgess: hi! Is there any next step about https://blueprints.launchpad.net/nova/+spec/libvirt-virtio-set-queue-sizes I can look at / help with? | |
| 14:04:35 | andreykurilin | superdan: sure | |
| 14:04:44 | superdan | andreykurilin: can you set the sort_keys=['uuid'] | |
| 14:05:23 | superdan | although that really shouldn't matter since we're only iterating instances in a single cell db here | |
| 14:05:40 | superdan | andreykurilin: and I can throw up a nova patch you can depends-on right? | |
| 14:06:09 | superdan | andreykurilin: got a bug number for this yet? | |
| 14:06:42 | andreykurilin | superdan: yes, we can do depends-on to check the fix. no, I do not have a bug report | |
| 14:08:13 | andreykurilin | superdan: superdan: made a patch to check sort_keys, but based on the queue of the zuul the results will be in several hours | |
| 14:11:01 | andreykurilin | superdan: should I create a bug report? | |
| 14:11:14 | superdan | andreykurilin: yeah please create a bug and I'll some debugging | |
| 14:11:19 | superdan | sorry, I'm stuck on a call atm | |