Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-04-09
22:12:24 dansmith "why the fsck are my computes requesting /foo/?bogus" would be high on my list
22:12:40 dansmith efried: I mean a legit check without spamming the logs in placement
22:12:59 dansmith a legit config-only check
22:13:05 efried one entry on startup is hardly spamming, but okay. auth_type it is, I guess.
22:13:41 dansmith I just restarted 1000 computes
22:13:55 imacdonn can we say (with reasonable confidence) that auth_type will always be a required option? TBH, I'm not sure why it doesn't default to 'password'
22:13:56 dansmith that's a lot of spam
22:14:15 efried This I cannot answer.
22:14:20 dansmith if I didn't, and I'm auditing logs, I find some /?bogus queries to placement and wonder who the fsck is on my network
22:14:29 dansmith and then realize it's ... mah compute nodes
22:14:34 dansmith I don't want to be on that customer call
22:15:06 efried imacdonn: In other places we go to load the auth from conf and error if it comes back None.
22:15:36 dansmith we call pre_start_hook before we even listen to RPC, and that calls update_available_resource, which will pull our RPs from placement
22:15:40 efried Which in this case might look a little bit like publicizing (bits of) SchedulerReportClient._create_client.
22:17:03 efried Nope, forget that.
22:18:06 efried So we could check for auth_type, and if not set, log a helpful warning, but not raise, and then let the failure happen organically down the line.
22:18:20 efried that's future-proof in the case where auth_type becomes not required.
22:18:27 dansmith so, we'd get the first failure before we even listen to RPC, run our first periodic, etc
22:24:58 imacdonn dansmith: so are you still advocating just removing the check entirely? I'd be cool with that, but mriedem may not be
22:25:19 dansmith imacdonn: if we can't do the check reliably based just what is in config, then yeah I think we should remove it
22:25:57 dansmith imacdonn: I also think that we're doing legit talking to placement so early in service startup, that we could mangle up some of that code to make it fail really hard if you really wanted to
22:26:10 dansmith but I also think that heavy logging of what we found when we reliably found it is completely legit
22:27:08 imacdonn dansmith: I guess my perception is that we want to fail hard, not just log something and continue anyway ..... perhaps because people only check logs if something is obviously not working (e.g. nova-compute service not coming up)
22:27:29 dansmith imacdonn: if compute can't report to placement, then things won't work
22:28:05 dansmith imacdonn: we do a similar thing because we're dependent on conductor.. we check at startup and log in a loop that we can't do anything
22:28:19 imacdonn dansmith: I've been led to believe that there's a scheduler config where it would work OK for the most part, but not report into placement
22:28:22 dansmith we have to do that late because we need RPC to be up before we can reliably know if conductor is up, just like this situation
22:28:43 dansmith imacdonn: if you're using cachingscheduler then you can avoid hitting placement in the scheduler
22:28:56 dansmith imacdonn: and you could argue that computes being misconfigured for placement would be "okay" in that situation
22:29:12 dansmith imacdonn: however, going forward, we *have* to migrate those people to having data in placement so we can get rid of that situation
22:29:19 imacdonn dansmith: right, I believe that that's the case that mriedem was concerned about
22:29:42 dansmith so that's a reason to hard fail, but again, if we can't reliably detect that except at runtime, then..we don't have much choice
22:30:11 dansmith imacdonn: what about this.. make the compute service self-disable if we fail to update our RPs in placement?
22:30:17 imacdonn I guess we can use auth_type ... It just feels a bit like making the same mistake a second time
22:30:20 dansmith we do that now for transient issues we find out about late, like libvirt stopping
22:30:27 dansmith we self-disable so that no new builds will come to us,
22:30:33 dansmith which is a big fat "I need help" to the operator
22:30:39 imacdonn interesting
22:30:45 dansmith and there's a reason field
22:30:57 dansmith self.disable(reason="you dun fscked up")
22:31:04 imacdonn heh
22:31:45 imacdonn I think I like that idea better
22:32:21 dansmith we also self-disable if too many builds fail for unknown reasons
22:32:39 imacdonn unit-testing for that could be "fun" ...
22:32:48 dansmith https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L1701-L1709
22:32:48 dansmith nah, 's easy
22:34:17 dansmith imacdonn: https://review.openstack.org/#/c/463597/
22:34:22 dansmith lots of test examples in there
22:34:30 mriedem just rely on auth_type https://review.openstack.org/#/q/status:open+project:openstack/nova+branch:master+topic:bug/1761487
22:34:35 mriedem we are doing that for neutron and cinder too
22:34:59 dansmith if that's legit, then sure
22:36:32 mriedem let me also state, for the record, that i don't care too much about this and said so awhile back http://eavesdrop.openstack.org/irclogs/%23openstack-nova/%23openstack-nova.2018-03-30.log.html#t2018-03-30T21:39:06
22:37:02 mriedem i definitely didn't think about the compute restart storm
22:38:18 mriedem and now i must attend to 1st grade homework
22:38:42 imacdonn efried: any objection to using auth_type ?
22:39:07 efried imacdonn: No. If they ever make that non-optional, I suspect many unintended side effects will result.
22:39:15 efried s/non-//
22:39:54 imacdonn yeah, and it seems like it'd probably become non-optional for neutron at the same time
22:41:25 efried Oh, it would be a ksa thing, so it would become optional for the world
22:41:52 imacdonn right, that's what I meant
22:42:33 imacdonn so if we do the same for placement as we do for neutron, such a change would break both at the same type
22:43:06 efried more or less. more or less.
22:45:55 imacdonn OK, I'll work on a new PS with that
22:46:02 imacdonn dansmith: thanks for the review
23:00:38 openstackgerrit Eric Berglund proposed openstack/nova master: PowerVM: Add proc_units_factor conf option https://review.openstack.org/554688
23:04:05 openstackgerrit Merged openstack/nova stable/queens: Add __repr__ for NovaException https://review.openstack.org/559158
#openstack-nova - 2018-04-10
01:21:32 openstackgerrit Merged openstack/nova master: Replace mox stubs with stub_out in test_extended_volumes.py https://review.openstack.org/558685
01:39:32 openstackgerrit jichenjc proposed openstack/nova master: WIP: remove ec2 in service and cmd https://review.openstack.org/556778
01:39:33 openstackgerrit jichenjc proposed openstack/nova master: WIP: remove db ec2 https://review.openstack.org/557572
01:39:33 openstackgerrit jichenjc proposed openstack/nova master: WIP: remove Ec2 object https://review.openstack.org/557150
01:39:46 openstackgerrit Jackie Truong proposed openstack/nova master: Implement certificate_utils https://review.openstack.org/479949
01:39:47 openstackgerrit Jackie Truong proposed openstack/nova master: Add trusted_image_certificates to REST API https://review.openstack.org/486204
01:41:50 openstackgerrit jichenjc proposed openstack/nova master: make metadata doc up to date https://review.openstack.org/559613
01:50:45 openstackgerrit Michael Still proposed openstack/nova master: Move xenapi disk resizing to privsep. https://review.openstack.org/552242
01:50:46 openstackgerrit Michael Still proposed openstack/nova master: Move xenapi partition copies to privsep. https://review.openstack.org/553605
01:50:46 openstackgerrit Michael Still proposed openstack/nova master: Sync xenapi and libvirt on what flags to pass e2fsck. https://review.openstack.org/554078
01:50:47 openstackgerrit Michael Still proposed openstack/nova master: We don't need utils.trycmd any more. https://review.openstack.org/554439
01:50:47 openstackgerrit Michael Still proposed openstack/nova master: Move image conversion to privsep. https://review.openstack.org/554437
01:50:48 openstackgerrit Michael Still proposed openstack/nova master: We no longer need rootwrap. https://review.openstack.org/554438
02:03:14 naichuans Hi, efried: do we have the scheduler of to get NRP working through GET /allocation_candidates?
02:05:34 alex_xu kashyap: yea, I think it supports boarder option, but at least for our team, we only use cmt
02:08:57 openstackgerrit Takashi NATSUME proposed openstack/nova master: Remove mox in test_virt_drivers.py https://review.openstack.org/559878
02:16:20 openstackgerrit Takashi NATSUME proposed openstack/nova master: Remove mox in unit/virt/xenapi/test_vm_utils.py (1) https://review.openstack.org/558704
03:43:22 openstackgerrit Yikun Jiang (Kero) proposed openstack/nova master: Add microversion to support extra_specs in flavor API. https://review.openstack.org/554159
03:51:25 openstackgerrit Takashi NATSUME proposed openstack/nova master: Remove mox in unit/virt/xenapi/test_vm_utils.py (2) https://review.openstack.org/558993
03:53:19 openstackgerrit Takashi NATSUME proposed openstack/nova master: Remove mox in unit/virt/xenapi/test_vm_utils.py (3) https://review.openstack.org/559258
04:00:15 openstackgerrit Yikun Jiang (Kero) proposed openstack/nova master: Add microversion to support extra_specs in flavor API. https://review.openstack.org/554159
05:39:27 openstackgerrit Naichuan Sun proposed openstack/nova master: xenapi: Support live migration in pooled multi-nodes environment https://review.openstack.org/489451
06:03:41 openstackgerrit jichenjc proposed openstack/nova master: remove ec2 object definitions https://review.openstack.org/557150
06:03:41 openstackgerrit jichenjc proposed openstack/nova master: remove ec2 in service and cmd https://review.openstack.org/556778
06:03:42 openstackgerrit jichenjc proposed openstack/nova master: remove ec2 db functions https://review.openstack.org/557572
06:25:16 openstackgerrit OpenStack Proposal Bot proposed openstack/nova master: Imported Translations from Zanata https://review.openstack.org/548772
06:45:09 openstackgerrit Naichuan Sun proposed openstack/nova master: xenapi: Support live migration in pooled multi-nodes environment https://review.openstack.org/489451
06:54:10 openstackgerrit Naichuan Sun proposed openstack/nova master: xenapi: Use XAPI pool instead of aggregate pool for shared SR migration https://review.openstack.org/554154
07:13:34 openstackgerrit Tetsuro Nakamura proposed openstack/nova-specs master: Return resources of entire trees in Placement https://review.openstack.org/559466
07:14:21 openstackgerrit Kashyap Chamarthy proposed openstack/nova master: libvirt: Bump MIN_{LIBVIRT,QEMU}_VERSION for "Rocky" https://review.openstack.org/558783
07:14:21 openstackgerrit Kashyap Chamarthy proposed openstack/nova master: Pick next minimum libvirt / QEMU versions for "Stein" https://review.openstack.org/558171
07:22:13 kashyap zigo: If you want to re-ACK this -- https://review.openstack.org/#/c/558171/
07:26:30 bauzas good morning nova
08:45:48 openstackgerrit Michael Still proposed openstack/nova master: Move xenapi disk resizing to privsep. https://review.openstack.org/552242

Earlier   Later