| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-02-02 | |||
| 18:33:17 | johnthetubaguy | jaypipes: thanks! | |
| 18:35:56 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Clarify the [cells] config option help https://review.openstack.org/428344 | |
| 18:35:57 | mriedem | melwitt: dansmith: ^ | |
| 18:37:06 | mriedem | EmilienM: that trace says the bdm is invalid | |
| 18:37:37 | EmilienM | bdm? | |
| 18:37:40 | mriedem | EmilienM: because the volume isn't found http://logs.openstack.org/15/424915/8/check/gate-puppet-openstack-integration-4-scenario001-tempest-centos-7/b29f35b/logs/nova/nova-compute.txt.gz#_2017-02-02_12_56_57_274 | |
| 18:37:45 | mriedem | bdm == block_device_mapping | |
| 18:37:48 | EmilienM | Block Device Mapping | |
| 18:37:49 | EmilienM | ok | |
| 18:38:01 | EmilienM | well, it worked yesterday | |
| 18:38:13 | mriedem | EmilienM: did it work with devstack? :P | |
| 18:38:19 | johnthetubaguy | mriedem: I can put something up for that bug 1659266 | |
| 18:38:19 | openstack | bug 1659266 in OpenStack Compute (nova) "Disk allocation for instance is not good with swap" [High,Confirmed] https://launchpad.net/bugs/1659266 - Assigned to Tanvir Talukder (tanvirt16) | |
| 18:38:26 | mriedem | johnthetubaguy: thanks | |
| 18:38:30 | EmilienM | mriedem: I was about to say it but well... | |
| 18:38:38 | johnthetubaguy | I have to run, soon, but lets see what can be done | |
| 18:38:50 | mriedem | johnthetubaguy: you're not allowed to leave | |
| 18:38:55 | EmilienM | mriedem: FYI we're delaying TripleO release because upgrades are not working currently from Newton to Ocata | |
| 18:38:55 | mriedem | sally will understand | |
| 18:39:07 | EmilienM | mriedem: we hope to solve blockers by next week :/ | |
| 18:39:14 | johnthetubaguy | mriedem: you haven't met the conductor, she is scary | |
| 18:41:15 | mriedem | EmilienM: my wild guess would be something related to https://review.openstack.org/#/c/427782/ | |
| 18:41:22 | mriedem | which merged last night | |
| 18:41:33 | mriedem | EmilienM: but would also mean you don't have things setup for cells v2 | |
| 18:42:07 | EmilienM | mriedem: do you run experimental job sometimes? | |
| 18:42:12 | EmilienM | it seems like you're not | |
| 18:42:44 | EmilienM | mriedem: I think we are pretty well covered for cells v2 but I'll need to confirm | |
| 18:43:00 | diana_clarke | EmilienM: I've run it, but I've yet to see it pass ;( | |
| 18:43:28 | EmilienM | diana_clarke: where? | |
| 18:43:48 | diana_clarke | EmilienM: for example: https://review.openstack.org/#/c/420186/ | |
| 18:44:10 | diana_clarke | (it's not recent, but people are running it) | |
| 18:44:52 | EmilienM | diana_clarke: cool | |
| 18:46:49 | dansmith | I run it too on occasion, but it takes a long time, fails often, and the logs are harder to sift through | |
| 18:47:03 | EmilienM | dansmith: ping me when it fails | |
| 18:47:06 | EmilienM | I'm happy to help | |
| 18:47:26 | openstackgerrit | John Garbutt proposed openstack/nova master: Stop swap allocations being wrong due to MB vs GB https://review.openstack.org/428352 | |
| 18:48:07 | EmilienM | dansmith: any idea about my bug mentioned above ^ | |
| 18:48:33 | dansmith | EmilienM: I looked a bit earlier, but I saw no debug logs to help track anything down | |
| 18:49:27 | EmilienM | dansmith: no debug on http://logs.openstack.org/15/424915/8/check/gate-puppet-openstack-integration-4-scenario001-tempest-centos-7/b29f35b/logs/nova/nova-compute.txt.gz#_2017-02-02_12_56_57_274 ? | |
| 18:49:32 | johnthetubaguy | jaypipes: mriedem: its a quick cut and refactor from the previous fix, hopefully, at least thats what the unit tests suggested: https://review.openstack.org/#/c/428352 | |
| 18:49:58 | EmilienM | dansmith: could it be related to cells again? | |
| 18:50:05 | dansmith | EmilienM: no, I was looking at another run that had no debug, periodic maybe? | |
| 18:50:21 | EmilienM | dansmith: yeah periodic but this one has debug ^ | |
| 18:51:24 | dansmith | EmilienM: ah, I hadn't seen that volume error, and no that's not cells related AFAIK | |
| 18:51:57 | openstackgerrit | John Garbutt proposed openstack/nova master: Stop swap allocations being wrong due to MB vs GB https://review.openstack.org/428352 | |
| 18:52:05 | EmilienM | dansmith: could it be related to https://review.openstack.org/#/c/427782/ ? | |
| 18:52:55 | openstackgerrit | John Garbutt proposed openstack/nova master: DNM: hack ironic with resource providers https://review.openstack.org/426296 | |
| 18:53:44 | dansmith | EmilienM: not the volume error for sure no | |
| 18:54:31 | EmilienM | well, if you have any idea so far, please let me know. We're stuck. | |
| 18:56:19 | openstackgerrit | Maciej Szankin proposed openstack/nova master: conf: add min parameter to scheduler opts https://review.openstack.org/428355 | |
| 18:56:57 | mriedem | EmilienM: so the instance not found you were talking about is like this http://logs.openstack.org/15/424915/8/check/gate-puppet-openstack-integration-4-scenario001-tempest-centos-7/b29f35b/console.html#_2017-02-02_13_04_48_204185 | |
| 18:57:03 | mriedem | in a bunch of the tempest tests | |
| 18:57:47 | mriedem | it looks like it's waiting for the server to be active, it's gone | |
| 18:58:01 | amoralej | EmilienM, dansmith, reported in https://bugs.launchpad.net/nova/+bug/1661360 | |
| 18:58:01 | openstack | Launchpad bug 1661360 in OpenStack Compute (nova) "tempest test fails with "Instance not found" error" [Undecided,New] | |
| 18:58:04 | jroll | johnthetubaguy: there's an experimental job for resource class stuff, so I was going to abandon https://review.openstack.org/#/c/426296/ | |
| 18:58:11 | jroll | johnthetubaguy: but can keep it around if you rather | |
| 18:58:42 | mriedem | EmilienM: i can put up a debug patch if you guys can depend on it | |
| 18:58:48 | amoralej | mriedem, i've reproduced locally, nova server returns "instance not found" but it's found in "nova list" command | |
| 18:59:04 | mriedem | amoralej: that's probably a race between when we have a build request and the actual instance, | |
| 18:59:07 | mriedem | the api returns either | |
| 18:59:15 | mriedem | since they both represent a server at different stages of the lifecycle | |
| 18:59:22 | EmilienM | mriedem: yes I can | |
| 18:59:29 | EmilienM | mriedem: you can run check experimental | |
| 19:00:48 | dansmith | mriedem: what race? | |
| 19:01:02 | melwitt | mriedem: isn't that issue fixed by https://review.openstack.org/#/c/427775 this morning? | |
| 19:01:22 | dansmith | melwitt: there's no delete going on here, which is what ^ is for right? | |
| 19:01:31 | melwitt | oh. yeah the above is for delete | |
| 19:04:08 | openstackgerrit | Matt Riedemann proposed openstack/nova master: WIP: debugging for instance not found in tripleo https://review.openstack.org/428360 | |
| 19:04:09 | mriedem | EmilienM: amoralej: ^ | |
| 19:04:42 | cdent | edleafe: do you recall if this allocation checking is happening anywhere else because this seems important but is stuck behind something else: https://review.openstack.org/#/c/416752/ | |
| 19:05:07 | mriedem | melwitt: dansmith: even with my fix, | |
| 19:05:31 | mriedem | couldn't we have a window of time where the build request exists, the instance exists, but the instance mapping doesn't? | |
| 19:05:44 | dansmith | mriedem: sure, but then we return the buildrequest, that's the point | |
| 19:05:53 | dansmith | mriedem: in their logs, I do not see "Found 1 cells" | |
| 19:05:54 | edleafe | cdent: it's not really "stuck"; just hasn't gotten core attention | |
| 19:06:35 | dansmith | mriedem: I wonder if it's the change from melwitt that skips looking in all cells -- maybe incorrectly following that path? | |
| 19:06:39 | cdent | edleafe: what I meant was that fix is +W and something we ought to release, but the other thing less so | |
| 19:06:41 | mriedem | dansmith: so maybe it hits a window when the api code sees there is no instance mapping, but then by the time it pulls the build request, that's gone | |
| 19:06:44 | edleafe | but yeah, it's not technically dependent on the one below it | |
| 19:06:56 | mriedem | that would be tight | |
| 19:07:01 | dansmith | mriedem: the api isn't even looking at the cells, though | |
| 19:07:17 | melwitt | that's only if CONF.cells.enable is True though | |
| 19:07:19 | edleafe | cdent: the other thing is as important | |
| 19:07:24 | dansmith | melwitt: I know, but | |
| 19:07:31 | edleafe | the create() method allows all sorts of bad stuff in | |
| 19:07:34 | EmilienM | mriedem: "check experimental" and we'll see what happens | |
| 19:07:46 | amoralej | I'm testing locally | |
| 19:07:49 | dansmith | melwitt: where is this in their log? https://github.com/openstack/nova/blob/master/nova/compute/api.py#L2518 | |
| 19:08:40 | cdent | edleafe: but it's not wired into the api is it? that's what I'm trying to get at. the patch above prevents the api being used badly, the patch below removes something that was never exposed | |
| 19:08:48 | dansmith | cells.enable=False in their startup dump | |
| 19:09:18 | melwitt | hm | |
| 19:09:29 | dansmith | melwitt: see what I mean though? | |
| 19:09:36 | melwitt | yeah | |
| 19:10:05 | dansmith | compare to a devstack run: http://logs.openstack.org/03/427903/1/check/gate-tempest-dsvm-neutron-full-ubuntu-xenial/fb34df9/logs/screen-n-api.txt.gz#_2017-02-01_21_00_47_552 | |
| 19:10:24 | edleafe | cdent: true | |
| 19:10:36 | edleafe | cdent: but rebasing on master will lose the +W | |
| 19:10:49 | dansmith | now it's possible that they have never had it right, and removing those pre-cellsv2 short circuits are why they're failing now, but that's exactly why those have to be gone | |
| 19:10:50 | melwitt | dansmith: I'm also wondering about your change that changed the fallbacks into raises of InstanceNotFound | |
| 19:10:51 | cdent | edleafe: so the question is: how much of a bug is it that it is possible to write negative allocations? | |