| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-07-28 | |||
| 15:59:16 | mriedem | there is a req-<uuid> thing | |
| 15:59:19 | s-dean | nova.osapi_compute.wsgi.server [req-b884b02e-e9c0-4845-9dd8-1f4cd1a6215a 2456405c5d3547e5aeece36828578f99 be5ea0b1031d4949ab94107522f305b4 - default default] 10.30.0.2 "GET /v2.1/servers/detail HTTP/1.1" status: 500 len: 566 time: 600.7115021 | |
| 15:59:20 | mriedem | you can follow that through the logs | |
| 15:59:26 | mriedem | req-b884b02e-e9c0-4845-9dd8-1f4cd1a6215a | |
| 15:59:27 | mriedem | yeah | |
| 15:59:32 | mriedem | so something exploded | |
| 15:59:34 | mriedem | 500 error | |
| 15:59:38 | mriedem | look for errors | |
| 15:59:52 | s-dean | DBError: Can't reconnect until invalid transaction is rolled back | |
| 16:00:18 | mriedem | huh, don't know what that is | |
| 16:00:28 | s-dean | its asking me to report a bug | |
| 16:00:36 | mriedem | s-dean: are you hand-rolling all this stuff or using some deployment tooling? | |
| 16:00:45 | s-dean | hand-rolling baby | |
| 16:00:47 | s-dean | haha | |
| 16:00:48 | mriedem | misconfiguration of the db wouldn't be a nova bug | |
| 16:00:57 | s-dean | probably a config error again | |
| 16:00:59 | mriedem | may i suggest using a deployment tool | |
| 16:01:17 | s-dean | yeah but which oone | |
| 16:01:17 | mriedem | like openstack-ansible or kolla | |
| 16:01:24 | mriedem | ^ are the ones i'd look at | |
| 16:01:25 | s-dean | i was looking into kolla | |
| 16:01:39 | s-dean | deploys it onto docker right | |
| 16:02:21 | mriedem | yeah | |
| 16:02:29 | mriedem | openstack-ansible deploys into lxc containers | |
| 16:02:39 | leakypipes | builds openstack as docker images is what you mean, but yeah. | |
| 16:03:01 | s-dean | ive just spent so much time learning how to deploy to bare metal | |
| 16:03:10 | s-dean | 3 node setup | |
| 16:03:26 | s-dean | i dont even now how all that stuff ties into my backend network | |
| 16:04:10 | s-dean | i wanted to deploy a self service network with overlay network provider and management hence why ive been doing it by hand | |
| 16:05:06 | s-dean | been one heck of a learning experience i'm glad its the weekend though hahaha | |
| 16:10:00 | openstackgerrit | Jay Pipes proposed openstack/nova master: remove source provider allocs in confirm_resize() https://review.openstack.org/488510 | |
| 16:11:51 | leakypipes | mriedem, superdan, bauwser, cdent, figleaf: ^^ | |
| 16:12:25 | leakypipes | that didn't mock everything out.. | |
| 16:12:26 | mriedem | leakypipes: we do-ish | |
| 16:12:39 | leakypipes | mriedem: see my qualifying statement above :) | |
| 16:12:53 | mriedem | like what? | |
| 16:13:08 | mriedem | we mock out cinder/glance/neutron yeah | |
| 16:13:14 | mriedem | but we don't really need those for testing this | |
| 16:13:34 | mriedem | e.g https://review.openstack.org/#/c/487958/ | |
| 16:15:09 | openstackgerrit | Merged openstack/nova master: libvirt: Post-migration, set cache value for Cinder volume(s) https://review.openstack.org/485752 | |
| 16:15:24 | leakypipes | mriedem: yeah, that's a good start. thanks for the link. | |
| 16:16:09 | leakypipes | mriedem: still no way to test the kinds of timing issues that superdan was bringing up on the call yesterday though... emulating a long MQ interaction on the destination for instance and ensuring that the source host does the right thing... | |
| 16:16:19 | leakypipes | anyway, I'm just bitching. it's Friday, deal with it :P | |
| 16:16:32 | mriedem | leakypipes: you're about as sassy as that gd sax | |
| 16:17:31 | melwitt | *sniff* my precious blueprint | |
| 16:17:50 | mriedem | sorry | |
| 16:18:32 | melwitt | s'okay :) | |
| 16:18:50 | mriedem | tell your bp to have a kick grass summer and you'll see it in the fall | |
| 16:18:59 | mriedem | in it's yearbook of course | |
| 16:19:10 | melwitt | haha | |
| 16:19:14 | mriedem | ps bobby flynn is soooo cute! | |
| 16:19:35 | melwitt | "Stay cool!" | |
| 16:27:31 | leakypipes | mriedem: grr, shitbuckets. | |
| 16:27:39 | leakypipes | mriedem: we forgot about resize to same host. :( | |
| 16:27:41 | mriedem | literally? | |
| 16:28:06 | leakypipes | mriedem: so when doing a resize same host, we'll end up deleting the entire allocation except for the shared providers in the patch I just put up. | |
| 16:28:29 | cdent | yeehaw | |
| 16:28:41 | mriedem | just check instance.host == CONF.host? | |
| 16:29:12 | leakypipes | mriedem: yeah. though... in the thing we added for scheduler claiming a double-up allocation, did we account for resize to same host? :( | |
| 16:29:49 | mriedem | yes | |
| 16:29:53 | mriedem | it comares the rp uuid right? | |
| 16:29:56 | mriedem | does a set difference | |
| 16:30:15 | mriedem | pretty sure i thought about that when reviewing and came to the conclusion it's handled by the set difference on rp uuid | |
| 16:30:28 | cdent | yeah, but isn’t the point that you still need a doubling and since the rp uuid is the same, you need to double within the allocation, not adjacent to | |
| 16:30:38 | mriedem | no | |
| 16:30:39 | leakypipes | mriedem: yeah, but the issue is that we need to create an allocation for the *same provider* but the resource amounts are added together (old and new amount) | |
| 16:30:44 | mriedem | the double is for the source and target | |
| 16:30:49 | superdan | yes | |
| 16:30:51 | mriedem | so you don't lose the source when putting allocations for the target | |
| 16:30:53 | superdan | but if resize to same host, you still have two copies | |
| 16:30:59 | superdan | so you need twice the allocation | |
| 16:31:06 | openstackgerrit | OpenStack Proposal Bot proposed openstack/nova master: Updated from global requirements https://review.openstack.org/488034 | |
| 16:31:26 | mriedem | i guess you do need to account for the resize up flavor resource class stuff | |
| 16:31:26 | leakypipes | superdan: well, not twice the allocation, but the new amount of resources added to the old amount of resources, but on the same resource provider. :( | |
| 16:31:46 | leakypipes | I hate my life. | |
| 16:31:47 | superdan | leakypipes: yes, I mean the sum of the allocations of course :) | |
| 16:31:48 | mriedem | so is it just that we aren't doing the correct 'new' flavor allocation when resize to same host? | |
| 16:31:56 | superdan | leakypipes: I've been using "doubled up" to mean "the sum" | |
| 16:31:56 | mriedem | is it sum? | |
| 16:32:04 | mriedem | for resize to same host i mean | |
| 16:32:10 | leakypipes | superdan: understood. | |
| 16:32:23 | superdan | it's definitely sum for disk, but I don't think we should distinguish, we should just sum everything | |
| 16:32:28 | mriedem | ok | |
| 16:32:30 | leakypipes | mriedem: yeah, it's sum | |
| 16:32:33 | mriedem | so let's open a bug? | |
| 16:32:42 | mriedem | tag it with pike-rc-potential | |
| 16:32:48 | superdan | if you're resizing to a new flavor with more disk, but fewer cpu/mem, you have to protect the older larger amount of resource | |
| 16:32:57 | mriedem | superdan: good point | |
| 16:33:44 | cdent | leakypipes: on the resourcce tracker side, when the resize confirm wants to delete everything on itself, would it be safe/okay to let that happen, and then immediately call into the _update routines so it would then write a new allocation for the correct size? | |
| 16:33:50 | leakypipes | ffs, we're just going to end up porting all the craziness and conditionals from the resource_tracker.py module into the scheduler report client for this stuff :( | |
| 16:34:49 | leakypipes | cdent: might be, yes... need to think through it. | |
| 16:35:06 | mriedem | cdent: _update only updates inventory | |
| 16:35:22 | mriedem | if you're talking about ResourceTracker._update | |
| 16:35:25 | mriedem | maybe we should rename that :) | |
| 16:35:27 | cdent | I don’t mean _update specifically say _update_* perhaps | |
| 16:35:31 | mriedem | heh | |
| 16:35:36 | cdent | wherever the regular allocations happens | |
| 16:35:50 | mriedem | _update_something_for_instance_maybe | |
| 16:36:06 | cdent | anyway, there’s another wrinkle near this that I wanted to ask about: are those regular allocation update shared providers aware yet? | |
| 16:36:25 | mriedem | don't think so | |