Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2017-08-01
21:44:07 melwitt I found one email to os-dev but it had no link to voting :P
21:44:25 melwitt vote ... somewhere ... that you should find by googling
21:45:23 edleafe mriedem: I'm consistent :)
21:54:29 cfriesen mriedem: for https://review.openstack.org/#/c/339715 would you be okay with me updating the "stale" migration to a "failed" or "error" state when we detect it? Or just check for them at nova-compute startup and assume they can't happen during "normal" running?
21:57:53 openstackgerrit Jackie Truong proposed openstack/nova master: Add trusted_certs to Instance object https://review.openstack.org/489408
21:58:03 jaypipes mriedem: marked that one patch -2.
22:00:07 cdent mriedem: does https://bugs.launchpad.net/nova/+bug/1708031 need an owner? If so, I can take it. Do we need the report client to be happy with both the old and new messages? Or just new? I’m guessing both.
22:00:07 openstack Launchpad bug 1708031 in OpenStack Compute (nova) ""Failed to delete inventory for resource provider" error messages for predictable 409 cases" [Medium,Triaged] - Assigned to Matt Riedemann (mriedem)
22:01:11 cdent mriedem: nm, you assigned it to yourself while I was reading it
22:01:40 openstackgerrit Matt Riedemann proposed openstack/nova master: Fix 409 handling in report client when deleting inventory https://review.openstack.org/489763
22:01:41 mriedem cdent: ^
22:07:53 cdent mriedem: has ironic been spewing these errors since early june?
22:08:09 openstackgerrit Michael Still proposed openstack/nova master: Move execs of tee to privsep. https://review.openstack.org/489438
22:08:09 openstackgerrit Michael Still proposed openstack/nova master: Read from console ptys using privsep. https://review.openstack.org/489486
22:08:20 zzzeek jaypipes: there?
22:08:40 mriedem cdent: probably
22:08:51 mriedem cdent: i only check the ironic ci job logs for ironic related chagnes
22:08:52 cdent mriedem: my real question: how come this didn’t break stuff ?
22:08:52 mriedem *changes
22:09:26 zzzeek mriedem: does nova get DB connection URLs from anywhere other than /etc/nova/nova.conf ?
22:10:23 mriedem zzzeek: yes
22:10:27 mriedem from the db for the cell mappings
22:10:31 zzzeek mriedem: in the DB !
22:10:33 mriedem nova_api.cell_mappings table
22:10:34 zzzeek mriedem: omg.
22:10:49 zzzeek mriedem: so. once you deploy nova, you can never change the ip number of the database ??
22:10:59 zzzeek ip number gets written into the DB itself ?
22:11:19 mriedem it's whatever is put in when creating the cell mapping
22:11:34 melwitt you can update the cell mapping
22:11:38 melwitt via nova-manage
22:11:56 melwitt if you need to change it
22:11:59 zzzeek melwitt: you can! did anyone consider maybe looking in the nova.conf file and seeing, oh hey they moved their DB server
22:12:08 zzzeek wow.
22:12:10 zzzeek nova. heh.
22:12:16 zzzeek OK
22:12:44 mriedem your volume connection info to a ceph server is also stored in a json blob in the db, so if your ceph server ip changes you have similar problems
22:12:51 mriedem per instance
22:12:58 mriedem so you have to migrate or reboot the instance
22:13:21 zzzeek WOW. OK wait, this actually destroys waht I want to do. wow
22:13:55 mriedem http://lists.openstack.org/pipermail/openstack-operators/2017-June/013700.html
22:14:26 zzzeek mriedem: im just looking at mysql urls
22:15:22 colby_ Hey Everyone I asked in the support channel but no one seemed to have an answer. We upgraded to newton and the flavor and aggregate migration scripts fail to cpoy the data to the nova_api tables. It appears that sqlalchecmy is adding the timezone offset to the DATETIME field and its complaining about incorrect format. Here is an example of the error:
22:15:23 colby_ Incorrect datetime value: '2015-02-17 22:37:06+00:00' for column 'created_at'
22:15:35 zzzeek mriedem: so confirm, it is completely impossible to have a scenario where a nova server on a particular controller had its own MySQL URL distinct from that of other contrlers, because they are pulled from the database and do not distinguish between the source
22:16:02 colby_ we use Mariadb 10.0.21
22:17:03 dansmith zzzeek: I don't know what you mean by "between the source" but yes, all the control services find the cell databases by looking in the api database
22:17:10 zzzeek dansmith: OK
22:18:08 melwitt each controller has to know how to connect to each of N cell databases, that's why they're in the database, there are multiple DB URLs we need to be able to connect to
22:22:25 mriedem melwitt: +2 on your detach shenanigans
22:23:20 melwitt \o/
22:25:33 melwitt mriedem: do you want me to move the comment inside the conditional? I can do that quick if you want to re +2
22:25:50 mriedem meh
22:25:56 melwitt haha, okay
22:26:09 mriedem my ability to make any sort of decision right now is very low
22:27:11 efried morgan mordred sdague https://review.openstack.org/#/c/488137/ is passing tests with temporary patches to twiddle glance & ironic confs.
22:27:19 mordred efried: woot!
22:27:45 efried morgan mordred sdague And FYI, PS5 has the swizzle where you pass the service_type (rather than the conf_group/project_name) to the util.
22:27:55 efried (which both sdague and morgan asked for)
22:27:57 morgan nice
22:28:17 efried Next stop: cinder!
22:29:27 mriedem huh, we're actually failing grenade at a decent clip http://logs.openstack.org/54/489754/3/check/gate-grenade-dsvm-neutron-multinode-ubuntu-xenial/7e19ce3/logs/old/screen-n-cond.txt.gz#_2017-08-01_21_57_19_559
22:29:38 mriedem on the ocata side
22:33:24 openstackgerrit Jackie Truong proposed openstack/nova master: Implement certificate_utils https://review.openstack.org/479949
22:35:20 cdent mriedem: in muiltinode does one node stay ocata? is this before or after that?
22:36:02 mordred efried: a few suggestions inline
22:37:26 mriedem cdent: depends on the job config
22:37:39 mriedem some keep n-cpu behind
22:38:19 mriedem but yeah i think multinode grenade always keeps n-cpu behind as n-1
22:38:22 mriedem and upgrades everything else
22:38:38 mriedem there used to be a single node grenade partial which upgraded everything except n-cpu
22:38:52 mriedem multinode grenade has 1 upgraded n-cpu and one n-1 n-cpu
22:39:06 mriedem so we can do things like live migrate between versions
22:39:18 zzzeek mriedem: crisis averted thanks for the help
22:41:29 openstackgerrit Chris Dent proposed openstack/nova master: WIP Always use application/json accept header in report client https://review.openstack.org/489772
22:41:51 mriedem zzzeek: can you shed some light on what the crisis was?
22:43:48 zzzeek mriedem: customer wants each controller accessing from its own source IP which is tied to a distinct password on the mysql server. because we use haproxy, that means each contorller must connect to a local haproxy, which means each controller has distinct database url
22:51:00 openstackgerrit Jackie Truong proposed openstack/nova master: Add trusted_certs to Instance object https://review.openstack.org/489408
22:51:42 openstackgerrit Jackie Truong proposed openstack/nova master: Implement certificate_utils https://review.openstack.org/479949
22:58:46 dansmith mriedem: people like that can just put the url in the database using an alias hostname, that they map per node in /etc/hosts
22:58:48 dansmith mriedem: easy stuff
22:59:10 dansmith mriedem: and it appears that they can also configure the creds in a mysql client config file that gets honored, so they can have separate creds too
22:59:46 dansmith which is way better than us trying to pattern-format the url we get out of the db or something silly like that
23:01:12 mriedem but can postgresql do that?!
23:29:38 gmann mriedem, re: API doc patches. not yet. i will push those today
23:40:40 gmann mriedem, can you restore this - https://review.openstack.org/#/c/489641/1
23:40:54 gmann api-microversion-history redirect is not done in https://review.openstack.org/#/c/489650/
23:42:12 mriedem gmann: that's weird
23:42:13 mriedem done
23:44:30 gmann mriedem, thanks, it might need rebase once 489650 is in. i will do after main patch is merged
23:49:52 openstackgerrit Ghanshyam Mann proposed openstack/nova master: Add redirect for api-microversion-history doc https://review.openstack.org/489641
23:54:31 openstackgerrit Matt Riedemann proposed openstack/nova master: Add redirect for api-microversion-history doc https://review.openstack.org/489641
#openstack-nova - 2017-08-02
00:16:33 openstackgerrit Takashi NATSUME proposed openstack/nova master: Enable cold migration with target host(1/2) https://review.openstack.org/408955
00:24:43 stewie925 hello nova gurus, quick question? is there any rules about special characters in, say, flavor name or tenant name, etc?
00:29:44 openstackgerrit Michael Still proposed openstack/nova master: Avoid chowning console logs in libvirt https://review.openstack.org/472229
00:29:45 openstackgerrit Michael Still proposed openstack/nova master: First attempt at adding a privsep user to nova itself. https://review.openstack.org/459166
00:29:45 openstackgerrit Michael Still proposed openstack/nova master: Move execs of touch to privsep. https://review.openstack.org/489190
00:29:46 openstackgerrit Michael Still proposed openstack/nova master: Move execs of tee to privsep. https://review.openstack.org/489438
00:29:46 openstackgerrit Michael Still proposed openstack/nova master: Move libvirt usages of chown to privsep. https://review.openstack.org/471972
00:29:47 openstackgerrit Michael Still proposed openstack/nova master: Read from console ptys using privsep. https://review.openstack.org/489486
00:29:47 openstackgerrit Michael Still proposed openstack/nova master: Refactor libvirt.utils.execute() away. https://review.openstack.org/489816

Earlier   Later