| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-01-23 | |||
| 16:40:02 | mriedem | incoming | |
| 16:40:15 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Set server status to ERROR if rebuild failed https://review.openstack.org/536268 | |
| 16:40:16 | bauzas | who's doing that ? I can take my pen if nobody does | |
| 16:40:32 | bauzas | oh, a new rev | |
| 16:40:36 | bauzas | mriedem: on it | |
| 16:41:07 | bauzas | mriedem: +2 | |
| 16:41:22 | bauzas | I can fast-approve or dansmith can +W | |
| 16:43:30 | mriedem | stvnoyes: ildikov: first victim of the multiattach swap volume test http://logs.openstack.org/89/532689/5/check/nova-multiattach/094e46b/job-output.txt.gz#_2018-01-23_12_43_58_790340 | |
| 16:44:59 | mriedem | this is where the swap starts http://logs.openstack.org/89/532689/5/check/nova-multiattach/094e46b/job-output.txt.gz#_2018-01-23_12_43_59_312226 | |
| 16:45:12 | openstackgerrit | melanie witt proposed openstack/nova stable/pike: Set server status to ERROR if rebuild failed https://review.openstack.org/536897 | |
| 16:45:44 | dmsimard | mriedem, kashyap: hi o/ It would be nice if we could land these in time for queens: https://review.openstack.org/#/c/524748/ && https://review.openstack.org/#/c/526748/ (and understand why the guestfs.set_backend_settings patch is breaking things when testing it here https://review.openstack.org/#/c/524750/ ) | |
| 16:46:06 | dmsimard | I suppose these are stable backport candidates as well | |
| 16:47:41 | mriedem | ildikov: stvnoyes: this is on a rax-dfw node and those have been really slow lately, so maybe it's just related to a slow node timeout since the swap volume tests do take awhile | |
| 16:48:04 | mriedem | dmsimard: i can't really say why things don't work with that stuff, | |
| 16:48:12 | mriedem | need some more libguestfs/qemu people | |
| 16:48:48 | dmsimard | mriedem: I don't know these people, I'm a mere mortal :( | |
| 16:49:12 | mriedem | dmsimard: i think https://review.openstack.org/#/c/526748/ is pretty straight forward though - don't do logging in a thread | |
| 16:49:17 | mriedem | b/c eventlet can explode | |
| 16:51:36 | stvnoyes | mriedem, ildikov, I finally got my new env working but I am seeing some failures in the 'normal' tempest multiattach tests. I am looking to see why that is. It is likely something unique to my env. that's what I'm doing atm. | |
| 16:52:03 | openstackgerrit | Jackie Truong proposed openstack/nova master: Add trusted_image_certificates to REST API https://review.openstack.org/486204 | |
| 16:55:24 | mriedem | it's doing the swap here http://logs.openstack.org/89/532689/5/check/nova-multiattach/094e46b/logs/screen-n-cpu.txt.gz#_Jan_23_11_16_02_654135 | |
| 16:56:55 | mriedem | sdague: do you ever remember anything about losing a requestid when using an elevated context? or people talking about resolving that? | |
| 16:57:17 | mriedem | i was tracing a swap volume operation through the logs and lose some of the logging once the context gets elevated | |
| 16:58:29 | openstackgerrit | Jackie Truong proposed openstack/python-novaclient master: Microversion 2.61 - Add trusted_image_certificates https://review.openstack.org/500396 | |
| 16:58:30 | openstackgerrit | Radoslav Gerganov proposed openstack/nova master: fix _ensure_resource_provider() https://review.openstack.org/536902 | |
| 17:01:49 | openstackgerrit | Radoslav Gerganov proposed openstack/nova master: WIP: VMware: implement update_provider_tree() https://review.openstack.org/536348 | |
| 17:03:09 | openstackgerrit | melanie witt proposed openstack/nova stable/ocata: Set server status to ERROR if rebuild failed https://review.openstack.org/536904 | |
| 17:13:05 | openstackgerrit | Chris Dent proposed openstack/nova master: WIP: Refactor WSGI apps and utils to limit imports https://review.openstack.org/533797 | |
| 17:15:56 | gibi | mriedem: btw, do you need me to run the weekly nova meeting on Thrusday? | |
| 17:16:28 | mriedem | gibi: maybe... | |
| 17:16:36 | mriedem | i'll let you know in a bit | |
| 17:16:47 | gibi | mriedem: OK, I can do that if needed | |
| 17:17:20 | mriedem | gibi: i should be able to do it, assuming i can remember :) | |
| 17:17:34 | gibi | mriedem: OK | |
| 17:36:49 | artom | melwitt, no newton backport for the rebuild imageref fix? There was some confusion whether newton is EOL or not | |
| 17:38:46 | dansmith | artom: it's EOL, just not deleted yet | |
| 17:38:50 | melwitt | artom: indeed the stable/newton branch is still around coincidentally. but earlier mriedem said it's probably not worth another newton release for this | |
| 17:39:44 | dansmith | I agree | |
| 17:40:25 | artom | Sure, not gonna argue with that | |
| 17:40:45 | artom | (I mean, I probably would, in theory) | |
| 17:40:58 | artom | But... I'm also realistic | |
| 17:55:45 | lyarwood | melwitt: thanks for your review of https://review.openstack.org/#/c/523958/ btw, are you going to be able to hit it again today? | |
| 17:56:53 | melwitt | lyarwood: yes, will re-review today | |
| 17:57:20 | lyarwood | melwitt: cool, thanks | |
| 18:08:39 | openstackgerrit | melanie witt proposed openstack/nova master: libvirt: pass the mdevs when rebooting the guest https://review.openstack.org/533818 | |
| 18:22:40 | Spazmotic | Whew busy day | |
| 18:25:08 | Spazmotic | If any core's are able to spare some cycles today for a XenAPI review, would love it. I am getting ready to head to bed but let me know if you have any questions. https://review.openstack.org/#/c/533168/ | |
| 18:41:20 | ildikov | mriedem: stvnoyes: sorry, partially off today | |
| 18:47:55 | openstackgerrit | Jackie Truong proposed openstack/python-novaclient master: Microversion 2.61 - Add trusted_image_certificates https://review.openstack.org/500396 | |
| 18:54:48 | mriedem | ildikov: np, i reported a bug against cinder, seems to be intermittent timing issue anyway | |
| 19:00:33 | ildikov | mriedem: that does not sound good | |
| 19:00:55 | ildikov | mriedem: the timing issue I mean | |
| 19:01:13 | ildikov | mriedem: Tnx for looking into it and reporting the big | |
| 19:01:26 | ildikov | *bug | |
| 19:16:52 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/pike: Make sure that functional test triggered on sample changes https://review.openstack.org/536934 | |
| 19:19:13 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/ocata: Make sure that functional test triggered on sample changes https://review.openstack.org/536936 | |
| 19:36:34 | mriedem | alex_xu: remember to update the placement API reference docs with the new 'required' parameter to GET /allocation_candidates https://review.openstack.org/#/c/535642/ | |
| 19:36:41 | melwitt | I was looking at this bug and it looks like the auto-disabling of compute can be used to DoS the cloud https://bugs.launchpad.net/nova/+bug/1742102 | |
| 19:36:43 | openstack | Launchpad bug 1742102 in OpenStack Compute (nova) "Simple user can disable compute" [Undecided,Confirmed] | |
| 19:37:02 | mriedem | melwitt: dansmith and i talked with the person that reported it a couple of weeks ago | |
| 19:37:21 | mriedem | problem is they are getting a super generic error on the nova side (VirtualInterfaceException due to the vif plugging failing) | |
| 19:37:38 | mriedem | all that nova gets back from neutron is the vif plug failed event, with no details about why it failed | |
| 19:37:56 | mriedem | so in that case we don't know if it's specific to that host or what | |
| 19:38:22 | dansmith | which is normally a thing worth disabling the compute over | |
| 19:38:24 | mriedem | we could whitelist a set of exceptions that we ignore for the auto-disable, but that gets tricky | |
| 19:38:30 | melwitt | the latest comment demonstrates it can be done via OverQuota, which is something I think we could avoid | |
| 19:38:46 | mriedem | i think they said it wasn't a quota issue, | |
| 19:38:59 | mriedem | because we check port quota in the api before casting to compute to build the instance and the port, | |
| 19:39:09 | mriedem | the issue was someting to do with no more fixed IPs in the pool for the network backend | |
| 19:39:30 | mriedem | if there is a way to preemptively know about that fixed IP thing, yeah we could check for that ahead of time | |
| 19:40:32 | melwitt | agreed it's tricky. the latest comment is an example involving volume quota, if the volume quota is 10 and they try to create 100 instances, 90 fail and they got 9 computes disabled | |
| 19:41:05 | dansmith | we should totally be able to distinguish something like that and not count that against the compute | |
| 19:41:07 | mriedem | yeah i know we definitely don't check volume quota, even in the api | |
| 19:41:26 | mriedem | https://review.openstack.org/#/c/520122/ | |
| 19:41:27 | melwitt | it feels like maybe auto-disable shouldn't be the default, it looks like this is easy to exploit | |
| 19:42:33 | melwitt | I'm thinking of if a public cloud is running openstack, someone could do this and get all the computes disabled. unless I'm missing something | |
| 19:42:43 | dansmith | in a public cloud it's less likely | |
| 19:42:49 | dansmith | because a single successful boot will reset the counter | |
| 19:42:56 | dansmith | I mean less likely IMHO | |
| 19:43:42 | mriedem | the fact we have no limit on the multiple instance create is a problem since you can just say give me 1000 instances, which can cause other issues like the rpc call to select_destinations to timeout | |
| 19:43:56 | mriedem | but much harder to mess with that when i spec'ed up some ideas | |
| 19:44:11 | mriedem | https://review.openstack.org/#/c/510235/ | |
| 19:44:33 | dansmith | we could do something like record the user-id of the last failure and not count subsequent fails by a single user, although that would fail for single-tenant clouds (i.e. NFV) | |
| 19:46:18 | melwitt | thinking ... I think all you'd need is a burst of requests to pretty well ensure the counter won't reset because usually the scheduler will pack. but yeah, maybe it's not as easy to exploit in a public cloud as I was thinking | |
| 19:47:22 | dansmith | I'm definitely in favor of catching and excluding quota-type failures if and where we can | |
| 19:48:52 | melwitt | yeah, that one is at least obvious. so any OverQuota that came come from compute (neutron/cinder) | |
| 20:08:01 | mriedem | cellsv2 map_instances usage bug thingy https://bugs.launchpad.net/nova/+bug/1742649 | |
| 20:08:02 | openstack | Launchpad bug 1742649 in nova (Ubuntu) "map_instances default batch size is too small." [Undecided,New] | |
| 20:08:13 | mriedem | mapping 200K instances took a long time | |
| 20:10:06 | melwitt | that's a lot of instances | |
| 20:10:08 | mriedem | because it maps all of them in a single shot but in batches of 50 | |
| 20:13:42 | melwitt | urgh. so really it would be best to have it similar to what we have in archive_deleted_rows now where you can give the max batch size but also say --until-complete | |
| 20:14:07 | mriedem | that's what he asked for in there yeah | |
| 20:14:30 | mriedem | "Actually, it would be useful to be able to say map all the instances using this batch size, eg: nova-manage --config-file /etc/nova/nova.conf cell_v2 map_instances --cell_uuid <uuid> --max_count all --batch_size 50000 " | |
| 20:14:49 | melwitt | yeah, I was thinking that sounded familiar and saw we have it for archive_deleted_rows | |
| 20:15:11 | mriedem | the thing about archive_deleted_rows is you're going to run that many times throughout the life of the deployment | |
| 20:15:20 | mriedem | map_instances is a done time deal when you're upgrading to cells v2 | |
| 20:15:23 | mriedem | *one time | |
| 20:15:27 | melwitt | yeah | |
| 20:15:52 | melwitt | I dunno, it could be a low-hanging-fruit thing to do | |