Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-01-23
19:38:22 dansmith which is normally a thing worth disabling the compute over
19:38:24 mriedem we could whitelist a set of exceptions that we ignore for the auto-disable, but that gets tricky
19:38:30 melwitt the latest comment demonstrates it can be done via OverQuota, which is something I think we could avoid
19:38:46 mriedem i think they said it wasn't a quota issue,
19:38:59 mriedem because we check port quota in the api before casting to compute to build the instance and the port,
19:39:09 mriedem the issue was someting to do with no more fixed IPs in the pool for the network backend
19:39:30 mriedem if there is a way to preemptively know about that fixed IP thing, yeah we could check for that ahead of time
19:40:32 melwitt agreed it's tricky. the latest comment is an example involving volume quota, if the volume quota is 10 and they try to create 100 instances, 90 fail and they got 9 computes disabled
19:41:05 dansmith we should totally be able to distinguish something like that and not count that against the compute
19:41:07 mriedem yeah i know we definitely don't check volume quota, even in the api
19:41:26 mriedem https://review.openstack.org/#/c/520122/
19:41:27 melwitt it feels like maybe auto-disable shouldn't be the default, it looks like this is easy to exploit
19:42:33 melwitt I'm thinking of if a public cloud is running openstack, someone could do this and get all the computes disabled. unless I'm missing something
19:42:43 dansmith in a public cloud it's less likely
19:42:49 dansmith because a single successful boot will reset the counter
19:42:56 dansmith I mean less likely IMHO
19:43:42 mriedem the fact we have no limit on the multiple instance create is a problem since you can just say give me 1000 instances, which can cause other issues like the rpc call to select_destinations to timeout
19:43:56 mriedem but much harder to mess with that when i spec'ed up some ideas
19:44:11 mriedem https://review.openstack.org/#/c/510235/
19:44:33 dansmith we could do something like record the user-id of the last failure and not count subsequent fails by a single user, although that would fail for single-tenant clouds (i.e. NFV)
19:46:18 melwitt thinking ... I think all you'd need is a burst of requests to pretty well ensure the counter won't reset because usually the scheduler will pack. but yeah, maybe it's not as easy to exploit in a public cloud as I was thinking
19:47:22 dansmith I'm definitely in favor of catching and excluding quota-type failures if and where we can
19:48:52 melwitt yeah, that one is at least obvious. so any OverQuota that came come from compute (neutron/cinder)
20:08:01 mriedem cellsv2 map_instances usage bug thingy https://bugs.launchpad.net/nova/+bug/1742649
20:08:02 openstack Launchpad bug 1742649 in nova (Ubuntu) "map_instances default batch size is too small." [Undecided,New]
20:08:13 mriedem mapping 200K instances took a long time
20:10:06 melwitt that's a lot of instances
20:10:08 mriedem because it maps all of them in a single shot but in batches of 50
20:13:42 melwitt urgh. so really it would be best to have it similar to what we have in archive_deleted_rows now where you can give the max batch size but also say --until-complete
20:14:07 mriedem that's what he asked for in there yeah
20:14:30 mriedem "Actually, it would be useful to be able to say map all the instances using this batch size, eg: nova-manage --config-file /etc/nova/nova.conf cell_v2 map_instances --cell_uuid <uuid> --max_count all --batch_size 50000 "
20:14:49 melwitt yeah, I was thinking that sounded familiar and saw we have it for archive_deleted_rows
20:15:11 mriedem the thing about archive_deleted_rows is you're going to run that many times throughout the life of the deployment
20:15:20 mriedem map_instances is a done time deal when you're upgrading to cells v2
20:15:23 mriedem *one time
20:15:27 melwitt yeah
20:15:52 melwitt I dunno, it could be a low-hanging-fruit thing to do
20:16:19 mriedem i guess
20:16:25 mriedem i'm not sure i'd want to review it though
20:16:33 mriedem high hanging review fruit
20:17:02 melwitt I wasn't thinking it would be hard to review. maybe I'm underestimating it
20:18:46 mriedem the only thing you're underestimating is my laziness at the moment
20:19:09 melwitt heh. fair enough :)
20:19:35 mriedem i'll put something in the docs and someone else can wizbang up a fancy new batch-size option or whatever
20:21:11 melwitt okay. I'll mention it at the cells meeting and see if tssurya might be interested
20:21:24 mriedem alright. already threw it on the meeting agenda
20:21:26 mriedem in the bugs list
20:21:32 melwitt coolness
20:21:58 mriedem btw, have we heard any updates from belmiro on the great cellsv2ification at cern?
20:22:01 mriedem and how it's going?
20:22:48 melwitt I haven't heard anything yet. just implied from the fixes they've been doing for other nova-manage cell_v2 commands, that they've been working on it
20:23:26 melwitt I think they'll need an improvement to map_instances themselves lest it take hours to run, heh
20:23:49 melwitt if they haven't already gone through it
20:27:19 openstackgerrit Hongbin Lu proposed openstack/nova master: Handle RPC timeout on get_console_output https://review.openstack.org/525335
20:33:41 openstackgerrit Matt Riedemann proposed openstack/nova master: Expand on when you might want to set --max-count for map_instances https://review.openstack.org/536959
20:34:29 openstackgerrit Matt Riedemann proposed openstack/nova master: Expand on when you might want to set --max-count for map_instances https://review.openstack.org/536959
20:50:20 melwitt what is with "libvirtError: Cannot recv data: Connection reset by peer" on stable, it's been flipping out, and specifically on the cells v1 job
20:52:33 mriedem is libvirt crashing?
20:52:37 mriedem check the libvirtd logs?
20:52:54 mriedem Roamer`: done https://review.openstack.org/#/c/140733/
20:58:30 Roamer` mriedem, just got the e-mail, thanks! Looking at it...
21:01:08 tonyb mriedem: my plan is to do the double checking today and then ask infra.
21:01:19 tonyb mriedem: (re tagging nova newton-eol)
21:01:25 melwitt doesn't look like a crash but I'm not too familiar with libvirtd logging. I see a message '"kvm" accelerator not found' several times http://logs.openstack.org/97/536897/1/check/legacy-tempest-dsvm-cells/1f42e60/logs/libvirt/libvirtd.txt.gz#_2018-01-23_18_37_23_873
21:06:00 mriedem doesn't look like it
21:06:10 mriedem tonyb: ok
21:07:30 melwitt comparing libvirtd.log with master change
21:08:08 tonyb melwitt: I wouldn't think that was a problem in a dsvm test
21:08:57 melwitt I don't find the kvm accelerator message in the master change libvirtd log
21:09:55 mriedem melwitt: what's the virt_type in nova.conf?
21:09:57 mriedem should be qemu
21:10:22 mriedem note that the version of libvirt is probably different between stable and master job runs
21:10:27 mriedem we use the pike UCA in master
21:10:49 melwitt huh, the nova-cpu.conf is very empty. http://logs.openstack.org/97/536897/1/check/legacy-tempest-dsvm-cells/1f42e60/logs/etc/nova/nova-cpu.conf.txt.gz
21:10:55 melwitt but in nova.conf virt_type = qemu
21:22:35 melwitt tonyb: what do you mean? the kvm accelerator thing shouldn't hurt?
21:23:11 mriedem melwitt: the cellsv1 job is probably not running in superconductor mode,
21:23:16 mriedem so doesn't use nova-cpu.conf
21:23:40 melwitt oh, okay
21:24:07 tonyb melwitt: just that in dsvm I'd expect kvm accelerator to me missing (as we should be using qemu), but it it strange that it's even looking for it
21:24:22 melwitt bah, there's a legit fail in the functional test_server_group tests on stable too. working on that
21:24:38 melwitt tonyb: oh, I see
21:30:13 openstackgerrit Merged openstack/nova master: ProviderTree.populate_from_iterable https://review.openstack.org/520756
21:33:18 openstackgerrit Merged openstack/nova master: Replace curly quotes with straight quotes https://review.openstack.org/536727
21:34:23 melwitt curly quotes eh?
21:35:15 openstackgerrit Hongbin Lu proposed openstack/nova master: Handle RPC timeout on get_console_output https://review.openstack.org/525335
21:35:37 jackie-truong If any core reviewer has a chance to take a look at this patch, it would be very much appreciated: https://review.openstack.org/#/c/486204/
21:37:32 jackie-truong jaypipes was looking at the patch series, but he's out at a work conference at the moment. He had taken away +2 from its dependent patch (https://review.openstack.org/#/c/489408/) due to the patch series not being complete. But patch series is complete now :-)
21:38:15 jackie-truong Usage and testing instructions can be found here: https://etherpad.openstack.org/p/queens-nova-certificate-validation
21:38:35 jackie-truong And we also wrote a barbican-tempest-plugin test for it: https://review.openstack.org/#/c/515210/
21:46:48 openstackgerrit melanie witt proposed openstack/nova stable/pike: Set server status to ERROR if rebuild failed https://review.openstack.org/536897
21:46:48 openstackgerrit melanie witt proposed openstack/nova stable/pike: Fix false positive server group functional tests https://review.openstack.org/536981
22:03:19 openstackgerrit Peter Penchev proposed openstack/nova master: Add the Nova libvirt StorPool attachment driver. https://review.openstack.org/140733
22:05:37 mriedem dansmith: on alex_xu's 1.12 allocation candidate change, i assume you ran into the same sticky part here https://review.openstack.org/#/c/536083/4/nova/scheduler/client/report.py@1153
22:05:47 mriedem with assuming that the client is getting a 1.12 format request,
22:06:14 mriedem we could be in the middle of a server create / cold migrate where the format is actually 1.10, you upgrade and restart conductor, and now the client code assumes it's going to get a 1.12 but it's 1.10 and kablammo
22:06:27 mriedem via reschedule from the compute
22:06:40 mriedem because we don't have a versioned object facade for this payload
22:10:17 Roamer` mriedem, new patchset up; thanks a lot for the very useful comments!
22:10:58 Roamer` (I guess now we wait for 12 hours or so for the checks to pass...)

Earlier   Later