Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-01-23
18:47:55 openstackgerrit Jackie Truong proposed openstack/python-novaclient master: Microversion 2.61 - Add trusted_image_certificates https://review.openstack.org/500396
18:54:48 mriedem ildikov: np, i reported a bug against cinder, seems to be intermittent timing issue anyway
19:00:33 ildikov mriedem: that does not sound good
19:00:55 ildikov mriedem: the timing issue I mean
19:01:13 ildikov mriedem: Tnx for looking into it and reporting the big
19:01:26 ildikov *bug
19:16:52 openstackgerrit Matt Riedemann proposed openstack/nova stable/pike: Make sure that functional test triggered on sample changes https://review.openstack.org/536934
19:19:13 openstackgerrit Matt Riedemann proposed openstack/nova stable/ocata: Make sure that functional test triggered on sample changes https://review.openstack.org/536936
19:36:34 mriedem alex_xu: remember to update the placement API reference docs with the new 'required' parameter to GET /allocation_candidates https://review.openstack.org/#/c/535642/
19:36:41 melwitt I was looking at this bug and it looks like the auto-disabling of compute can be used to DoS the cloud https://bugs.launchpad.net/nova/+bug/1742102
19:36:43 openstack Launchpad bug 1742102 in OpenStack Compute (nova) "Simple user can disable compute" [Undecided,Confirmed]
19:37:02 mriedem melwitt: dansmith and i talked with the person that reported it a couple of weeks ago
19:37:21 mriedem problem is they are getting a super generic error on the nova side (VirtualInterfaceException due to the vif plugging failing)
19:37:38 mriedem all that nova gets back from neutron is the vif plug failed event, with no details about why it failed
19:37:56 mriedem so in that case we don't know if it's specific to that host or what
19:38:22 dansmith which is normally a thing worth disabling the compute over
19:38:24 mriedem we could whitelist a set of exceptions that we ignore for the auto-disable, but that gets tricky
19:38:30 melwitt the latest comment demonstrates it can be done via OverQuota, which is something I think we could avoid
19:38:46 mriedem i think they said it wasn't a quota issue,
19:38:59 mriedem because we check port quota in the api before casting to compute to build the instance and the port,
19:39:09 mriedem the issue was someting to do with no more fixed IPs in the pool for the network backend
19:39:30 mriedem if there is a way to preemptively know about that fixed IP thing, yeah we could check for that ahead of time
19:40:32 melwitt agreed it's tricky. the latest comment is an example involving volume quota, if the volume quota is 10 and they try to create 100 instances, 90 fail and they got 9 computes disabled
19:41:05 dansmith we should totally be able to distinguish something like that and not count that against the compute
19:41:07 mriedem yeah i know we definitely don't check volume quota, even in the api
19:41:26 mriedem https://review.openstack.org/#/c/520122/
19:41:27 melwitt it feels like maybe auto-disable shouldn't be the default, it looks like this is easy to exploit
19:42:33 melwitt I'm thinking of if a public cloud is running openstack, someone could do this and get all the computes disabled. unless I'm missing something
19:42:43 dansmith in a public cloud it's less likely
19:42:49 dansmith because a single successful boot will reset the counter
19:42:56 dansmith I mean less likely IMHO
19:43:42 mriedem the fact we have no limit on the multiple instance create is a problem since you can just say give me 1000 instances, which can cause other issues like the rpc call to select_destinations to timeout
19:43:56 mriedem but much harder to mess with that when i spec'ed up some ideas
19:44:11 mriedem https://review.openstack.org/#/c/510235/
19:44:33 dansmith we could do something like record the user-id of the last failure and not count subsequent fails by a single user, although that would fail for single-tenant clouds (i.e. NFV)
19:46:18 melwitt thinking ... I think all you'd need is a burst of requests to pretty well ensure the counter won't reset because usually the scheduler will pack. but yeah, maybe it's not as easy to exploit in a public cloud as I was thinking
19:47:22 dansmith I'm definitely in favor of catching and excluding quota-type failures if and where we can
19:48:52 melwitt yeah, that one is at least obvious. so any OverQuota that came come from compute (neutron/cinder)
20:08:01 mriedem cellsv2 map_instances usage bug thingy https://bugs.launchpad.net/nova/+bug/1742649
20:08:02 openstack Launchpad bug 1742649 in nova (Ubuntu) "map_instances default batch size is too small." [Undecided,New]
20:08:13 mriedem mapping 200K instances took a long time
20:10:06 melwitt that's a lot of instances
20:10:08 mriedem because it maps all of them in a single shot but in batches of 50
20:13:42 melwitt urgh. so really it would be best to have it similar to what we have in archive_deleted_rows now where you can give the max batch size but also say --until-complete
20:14:07 mriedem that's what he asked for in there yeah
20:14:30 mriedem "Actually, it would be useful to be able to say map all the instances using this batch size, eg: nova-manage --config-file /etc/nova/nova.conf cell_v2 map_instances --cell_uuid <uuid> --max_count all --batch_size 50000 "
20:14:49 melwitt yeah, I was thinking that sounded familiar and saw we have it for archive_deleted_rows
20:15:11 mriedem the thing about archive_deleted_rows is you're going to run that many times throughout the life of the deployment
20:15:20 mriedem map_instances is a done time deal when you're upgrading to cells v2
20:15:23 mriedem *one time
20:15:27 melwitt yeah
20:15:52 melwitt I dunno, it could be a low-hanging-fruit thing to do
20:16:19 mriedem i guess
20:16:25 mriedem i'm not sure i'd want to review it though
20:16:33 mriedem high hanging review fruit
20:17:02 melwitt I wasn't thinking it would be hard to review. maybe I'm underestimating it
20:18:46 mriedem the only thing you're underestimating is my laziness at the moment
20:19:09 melwitt heh. fair enough :)
20:19:35 mriedem i'll put something in the docs and someone else can wizbang up a fancy new batch-size option or whatever
20:21:11 melwitt okay. I'll mention it at the cells meeting and see if tssurya might be interested
20:21:24 mriedem alright. already threw it on the meeting agenda
20:21:26 mriedem in the bugs list
20:21:32 melwitt coolness
20:21:58 mriedem btw, have we heard any updates from belmiro on the great cellsv2ification at cern?
20:22:01 mriedem and how it's going?
20:22:48 melwitt I haven't heard anything yet. just implied from the fixes they've been doing for other nova-manage cell_v2 commands, that they've been working on it
20:23:26 melwitt I think they'll need an improvement to map_instances themselves lest it take hours to run, heh
20:23:49 melwitt if they haven't already gone through it
20:27:19 openstackgerrit Hongbin Lu proposed openstack/nova master: Handle RPC timeout on get_console_output https://review.openstack.org/525335
20:33:41 openstackgerrit Matt Riedemann proposed openstack/nova master: Expand on when you might want to set --max-count for map_instances https://review.openstack.org/536959
20:34:29 openstackgerrit Matt Riedemann proposed openstack/nova master: Expand on when you might want to set --max-count for map_instances https://review.openstack.org/536959
20:50:20 melwitt what is with "libvirtError: Cannot recv data: Connection reset by peer" on stable, it's been flipping out, and specifically on the cells v1 job
20:52:33 mriedem is libvirt crashing?
20:52:37 mriedem check the libvirtd logs?
20:52:54 mriedem Roamer`: done https://review.openstack.org/#/c/140733/
20:58:30 Roamer` mriedem, just got the e-mail, thanks! Looking at it...
21:01:08 tonyb mriedem: my plan is to do the double checking today and then ask infra.
21:01:19 tonyb mriedem: (re tagging nova newton-eol)
21:01:25 melwitt doesn't look like a crash but I'm not too familiar with libvirtd logging. I see a message '"kvm" accelerator not found' several times http://logs.openstack.org/97/536897/1/check/legacy-tempest-dsvm-cells/1f42e60/logs/libvirt/libvirtd.txt.gz#_2018-01-23_18_37_23_873
21:06:00 mriedem doesn't look like it
21:06:10 mriedem tonyb: ok
21:07:30 melwitt comparing libvirtd.log with master change
21:08:08 tonyb melwitt: I wouldn't think that was a problem in a dsvm test
21:08:57 melwitt I don't find the kvm accelerator message in the master change libvirtd log
21:09:55 mriedem melwitt: what's the virt_type in nova.conf?
21:09:57 mriedem should be qemu
21:10:22 mriedem note that the version of libvirt is probably different between stable and master job runs
21:10:27 mriedem we use the pike UCA in master
21:10:49 melwitt huh, the nova-cpu.conf is very empty. http://logs.openstack.org/97/536897/1/check/legacy-tempest-dsvm-cells/1f42e60/logs/etc/nova/nova-cpu.conf.txt.gz
21:10:55 melwitt but in nova.conf virt_type = qemu
21:22:35 melwitt tonyb: what do you mean? the kvm accelerator thing shouldn't hurt?
21:23:11 mriedem melwitt: the cellsv1 job is probably not running in superconductor mode,
21:23:16 mriedem so doesn't use nova-cpu.conf
21:23:40 melwitt oh, okay
21:24:07 tonyb melwitt: just that in dsvm I'd expect kvm accelerator to me missing (as we should be using qemu), but it it strange that it's even looking for it
21:24:22 melwitt bah, there's a legit fail in the functional test_server_group tests on stable too. working on that
21:24:38 melwitt tonyb: oh, I see
21:30:13 openstackgerrit Merged openstack/nova master: ProviderTree.populate_from_iterable https://review.openstack.org/520756
21:33:18 openstackgerrit Merged openstack/nova master: Replace curly quotes with straight quotes https://review.openstack.org/536727
21:34:23 melwitt curly quotes eh?

Earlier   Later